I'm trying to understand the best way to instrument the default thread pool executor used by AWS async client. Source. We are using Micrometer for tracking all metrics.
I aim to understand if I should use a custom thread pool executor or if the default one is good enough.
My current code is as follows (and it works as expected):
//use the same thread pool executor as the aws sdk but instrument it.
int processors = Runtime.getRuntime().availableProcessors();
int corePoolSize = Math.max(8, processors);
int maxPoolSize = Math.max(64, processors * 2);
ThreadPoolExecutor executor = new ThreadPoolExecutor(corePoolSize, maxPoolSize,
10, TimeUnit.SECONDS,
new LinkedBlockingQueue<>(1_000),
new ThreadFactoryBuilder()
.threadNamePrefix("sdk-async-response").build());
// Allow idle core threads to time out
executor.allowCoreThreadTimeOut(true);
var instrumentedExecutor = ExecutorServiceMetrics.monitor(meterRegistry,
executor,
"instrumented-ddb-executor");
return DynamoDbAsyncClient.builder()
.asyncConfiguration(b -> b.advancedOption(FUTURE_COMPLETION_EXECUTOR, instrumentedExecutor))
.build();
I feel there is a simpler way but cannot find it. Is there a simpler way to instrument without redefining the default executor?
You can override default metric collector for micrometer: this is my implementation: https://gist.github.com/PatrykGala/e4aec004eb55cd8cbdee328f217771c7
Related
I am building a device for my research team. To briefly describe it, this device uses a motor and load sensor connected to an Arduino to apply a rotational force to a corn stalk and record the resistance of the stalk. We are in the process of building Bluetooth into the device. We are using this BT module.
We have a BLE GATT Service with 2 characteristics for storing DATA and 1 for holding the command which is an integer that will be read by the device and acted on. Reading the command characteristic is where we encounter our problem.
void get_input(){
uint16_t bufSize = 15;
char inputBuffer[bufSize];
bleParse = Adafruit_ATParser(); // Throws error: bleParse was not declared in this scope
bleParse.atcommandStrReply("AT+GATTCHAR=3",&inputBuffer,bufSize,1000);
Serial.print("input:");
Serial.println(inputBuffer);
}
The functions I am trying to use are found in the library for the module in Adarfruit_ATParser.cpp
/******************************************************************************/
/*!
#brief Constructor
*/
/******************************************************************************/
Adafruit_ATParser::Adafruit_ATParser(void)
{
_mode = BLUEFRUIT_MODE_COMMAND;
_verbose = false;
}
******************************************************************************/
/*!
#brief Send an AT command and get multiline string response into
user-provided buffer.
#param[in] cmd Command
#param[in] buf Provided buffer
#param[in] bufsize buffer size
#param[in] timeout timeout in milliseconds
*/
/******************************************************************************/
uint16_t Adafruit_ATParser::atcommandStrReply(const char cmd[], char* buf, uint16_t bufsize, uint16_t timeout)
{
uint16_t result_bytes;
uint8_t current_mode = _mode;
// switch mode if necessary to execute command
if ( current_mode == BLUEFRUIT_MODE_DATA ) setMode(BLUEFRUIT_MODE_COMMAND);
// Execute command with parameter and get response
println(cmd);
result_bytes = this->readline(buf, bufsize, timeout, true);
// switch back if necessary
if ( current_mode == BLUEFRUIT_MODE_DATA ) setMode(BLUEFRUIT_MODE_DATA);
return result_bytes;
}
None of the examples in the library use this. They all create their own parsers. For example, the neopixel_picker example sketch has a file called packetParser.cpp which I believe retrieves data from the BT module for that specific sketch, but it never includes or uses Adafruit_ATParser.. There are no examples of this constructor anywhere and I cannot figure out how to use it. I have tried these ways:
bleParse = Adafruit_ATParser();
Adafruit_ATParser bleParse();
Adafruit_ATParser();
ble.Adafruit_ATParser bleParse();
note: ble is an object that signifies a Serial connection between arduino and BT created with:
SoftwareSerial bluefruitSS = SoftwareSerial(BLUEFRUIT_SWUART_TXD_PIN, BLUEFRUIT_SWUART_RXD_PIN);
Adafruit_BluefruitLE_UART ble(bluefruitSS, BLUEFRUIT_UART_MODE_PIN,BLUEFRUIT_UART_CTS_PIN, BLUEFRUIT_UART_RTS_PIN);
Can anyone give me a clue on how to use the Adafruit_ATParser() constructor? Also, if the constructor has no reference to the ble object, how does it pass AT commands to the BT module?
I know this is a big ask, I appreciate any input you can give me.
Like this
Adafruit_ATParser bleParse;
You were closest with this one Adafruit_ATParser bleParse();. This is a common beginner mistake because it looks right. Unfortunately it declares a function bleParse which takes no arguments and returns a Adafruit_ATParser object.
I can't answer the second question.
EDIT
I've taken the time to have a look at the code. This is what I found
class Adafruit_BluefruitLE_UART : public Adafruit_BLE
{
and
class Adafruit_BLE : public Adafruit_ATParser
{
what this means is that the Adafruit_BluefruitLE_UART class is derived from the Adafruit_BLE class which in turn is derived from the Adafruit_ATParser class. Derivation means that any public methods in Adafruit_BLE can also be used on a Adafruit_BluefruitLE_UART object. You already have an Adafruit_BluefruitLE_UART object (you called it ble) so you can just use the method you want to use on that object.
SoftwareSerial bluefruitSS = SoftwareSerial(BLUEFRUIT_SWUART_TXD_PIN, BLUEFRUIT_SWUART_RXD_PIN);
Adafruit_BluefruitLE_UART ble(bluefruitSS, BLUEFRUIT_UART_MODE_PIN,BLUEFRUIT_UART_CTS_PIN, BLUEFRUIT_UART_RTS_PIN);
ble.atcommandStrReply( ... );
How do you override the Hystrix default configuration for OpenFeign? Most of the documentation out there is for SpringBoot + OpenFeign, which has its own Spring-specific configuration override system.
Ideally it would be possible to configure the Hystrix core size for the client and configure and timeouts on a per endpoint basis.
Hystrix OpenFeign has a setterFactory() method on the builder that allows you to pass in a SetterFactory lambda function that is executed when setting up each target endpoint:
final SetterFactory hystrixConfigurationFactory = (target, method) -> {
final String groupKey = target.name();
final String commandKey = method.getAnnotation(RequestLine.class).value();
// Configure default thread pool properties
final HystrixThreadPoolProperties.Setter hystrixThreadPoolProperties = HystrixThreadPoolProperties.Setter()
.withCoreSize(50)
.withMaximumSize(200)
.withAllowMaximumSizeToDivergeFromCoreSize(true);
return HystrixCommand.Setter
.withGroupKey(HystrixCommandGroupKey.Factory.asKey(groupKey))
.andCommandKey(HystrixCommandKey.Factory.asKey(commandKey))
.andThreadPoolPropertiesDefaults(hystrixThreadPoolProperties);;
};
final MyTargetClient myTargetClient = HystrixFeign.builder()
.setterFactory(hystrixConfigurationFactory)
.client(new OkHttpClient())
.encoder(new JacksonEncoder(objectMapper))
.decoder(new JacksonDecoder(objectMapper))
.target(new Target.HardCodedTarget<>(MyTargetClient.class, "customclientname", baseUrl))
The above example uses boilerplate from the OpenFeign documentation to properly name Hystrix keys based on the target endpoint function. It then goes further by also configuring the default thread pool property core size and maximum core size as a default for all of the target functions.
However, since this factory is called for each target endpoint, we can actually override the Hystrix configuration on a per endpoint basis. A use good case for this is Hystrix timeouts: sometimes there are endpoints that take longer than others and we need to account for that.
The easiest way would be to first create an annotation and place it on the target endpoints that need to be overridden:
/**
* Override Hystrix configuration for Feign targets.
*/
#Retention(RetentionPolicy.RUNTIME)
#Target(ElementType.METHOD)
#interface HystrixOverride {
int DEFAULT_EXECUTION_TIMEOUT = 2_000;
/**
* Execution timeout in milliseconds.
*/
int executionTimeout() default DEFAULT_EXECUTION_TIMEOUT;
}
interface MyTargetClient {
#HystrixOverride(executionTimeout = 10_000)
#RequestLine("GET /rest/{storeCode}/V1/products")
Products searchProducts(#Param("storeCode") String storeCode, #QueryMap Map<String, Object> queryMap);
#RequestLine("GET /rest/{storeCode}/V1/products/{sku}")
Product getProduct(#Param("storeCode") String storeCode, #Param("sku") String sku);
}
In the above example, the search API might take a little longer to load so we have an override for that.
Just putting the override annotation on the target endpoint function is not enough though. We need to go back to our factory and update it to use the data in the annotations:
final SetterFactory hystrixConfigurationFactory = (target, method) -> {
final String groupKey = target.name();
final String commandKey = method.getAnnotation(RequestLine.class).value();
// Configure per-function Hystrix configuration by referencing annotations
final HystrixCommandProperties.Setter hystrixCommandProperties = HystrixCommandProperties.Setter();
final HystrixOverride hystrixOverride = method.getAnnotation(HystrixOverride.class);
final int executionTimeout = (hystrixOverride == null)
? HystrixOverride.DEFAULT_EXECUTION_TIMEOUT
: hystrixOverride.executionTimeout();
hystrixCommandProperties.withExecutionTimeoutInMilliseconds(executionTimeout);
// Configure default thread pool properties
final HystrixThreadPoolProperties.Setter hystrixThreadPoolProperties = HystrixThreadPoolProperties.Setter()
.withCoreSize(50)
.withMaximumSize(200)
.withAllowMaximumSizeToDivergeFromCoreSize(true);
return HystrixCommand.Setter
.withGroupKey(HystrixCommandGroupKey.Factory.asKey(groupKey))
.andCommandKey(HystrixCommandKey.Factory.asKey(commandKey))
.andCommandPropertiesDefaults(hystrixCommandProperties)
.andThreadPoolPropertiesDefaults(hystrixThreadPoolProperties);;
};
The above checks that an override annotation exists and then uses the data in that annotation to configure the execution timeout for that target endpoint. If the override is not present, the default for the HystrixOverride endpoint will be used instead. The resulting hystrixCommandProperties variable is then plugged in to the overall HystrixCommand.Setter at the end.
I have been trying to stream a data into MySQL database using APACHE KAFKA and FLUME. (Here is my flume configuration file)
agent.sources=kafkaSrc
agent.channels=channel1
agent.sinks=jdbcSink
agent.channels.channel1.type=org.apache.flume.channel.kafka.KafkaChannel
agent.channels.channel1.brokerList=localhost:9092
agent.channels.channel1.topic=kafkachannel
agent.channels.channel1.zookeeperConnect=localhost:2181
agent.channels.channel1.capacity=10000
agent.channels.channel1.transactionCapacity=1000
agent.sources.kafkaSrc.type = org.apache.flume.source.kafka.KafkaSource
agent.sources.kafkaSrc.channels = channel1
agent.sources.kafkaSrc.zookeeperConnect = localhost:2181
agent.sources.kafkaSrc.topic = kafka-mysql
***agent.sinks.jdbcSink.type = How to declare this?***
agent.sinks.jdbcSink.connectionString = jdbc:mysql://1.1.1.1:3306/test
agent.sinks.jdbcSink.username=user
agent.sinks.jdbcSink.password=password
agent.sinks.jdbcSink.batchSize = 10
agent.sinks.jdbcSink.channel =channel1
agent.sinks.jdbcSink.sqlDialect=MYSQL
agent.sinks.jdbcSink.driver=com.mysql.jdbc.Driver
agent.sinks.jdbcSink.sql=(${body:varchar})
I know how to stream data into hadoop or hbase (logger type or hdfs type), However can't find a type to stream into mysql DB. So my question is how do i declare the jdbcSink.type?
You could always create a custom sink for MySQL. This is what we did at FIWARE with Cygnus tool.
Feel free to get inspired from it: https://github.com/telefonicaid/fiware-cygnus/blob/master/cygnus-ngsi/src/main/java/com/telefonica/iot/cygnus/sinks/NGSIMySQLSink.java
It extends this other custom base class for all our sinks: https://github.com/telefonicaid/fiware-cygnus/blob/master/cygnus-ngsi/src/main/java/com/telefonica/iot/cygnus/sinks/NGSISink.java
Basically, you have to extend AbstractSink and implement the Configurable interface. That means to override al least the following methods:
public Status process() throws EventDeliveryException
and:
public void configure(Context context)
respectively.
I'm creating a back-end server application in Dart which is using a MySQL database to store data. To make the SQL call I'm using the ConnectionPool from SqlJocky.
What I do when the app starts:
Create a singleton which store the ConnectionPool
Execute multiple queries with prepareExecute and query
Locally this approach is working fine. Now I pushed a development version to Heroku and I'm getting connection issues after a few minutes.
So I wonder, do I need to close/release a single connection from the pool I use to execute a query? Or is the connection after the query placed again in the pool and free for use?
The abstract base class for all the MySQL stores:
abstract class MySQLStore {
MySQLStore(ConnectionPool connectionPool) {
this._connectionPool = connectionPool;
}
ConnectionPool get connectionPool => this._connectionPool;
ConnectionPool _connectionPool;
}
A concrete implementation for the method getAll:
Future<List<T>> getAll() async {
Completer completer = new Completer();
connectionPool.query("SELECT id, name, description FROM role").then((result) {
return result.toList();
}).then((rows) {
completer.complete(this._processRows(rows));
}).catchError((error) {
// TODO: Better error handling.
print(error);
completer.complete(null);
});
return completer.future;
}
The error I get:
SocketException: OS Error: Connection timed out, errno = 110, address = ...
This doesn't fully answer your question but I think you could simplify your code like:
Future<List<T>> getAll() async {
try {
var result = await connectionPool.query(
"SELECT id, name, description FROM role");
return this._processRows(await result.toList());
} catch(error) {
// TODO: Better error handling.
print(error);
return null;
}
}
I'm sure here is no need to close a connection with query. I don't know about prepareExecute though.
According to a comment in the SqlJocky code it can take quite some time for a connection to be released by the database server.
Maybe you need to increase the connection pool size (default 5) so you don't run out of connections while ConnectionPool is waiting for connections to be released.
After some feedback from Heroku I managed to resolve this problem by implementing a timer task that does every 50 seconds a basic MySQL call.
The response from Heroku:
Heroku's networking enforces an idle timeout of 60-90 seconds to prevent runaway processes. If you're using persistent connections in your application, make sure that you're sending a keep-alive at, say, 55 seconds to prevent your open connection from being dropped by the server.
The work around code:
const duration = const Duration(seconds: 50);
new Timer.periodic(duration, (Timer t) {
// Do a simple MySQL call with on the connection pool.
this.connectionPool.execute('SELECT id from role');
print('*** Keep alive triggered for MySQL heroku ***');
});
Ok, this question is where I reached after trying out some stuff. I'll first give a brief intro to what I wanted to do and how I got here.
I'm writing a script to start an EMR cluster using Java AWS SDK. The EMR cluster is to be started inside a VPC and a subnet with a certain id. When I specify the subnet id (code line below ending with // ******) the emr cluster stays in the STARTING state and does not move ahead for several minutes, eventually giving up and failing. I'm not sure if there's a bug with the implementation of this functionality in the SDK.
try {
/**
* Specifying credentials
*/
String accessKey = EmrUtils.ACCESS_KEY;
String secretKey = EmrUtils.SECRET_ACCESS_KEY;
AWSCredentials credentials = new BasicAWSCredentials(accessKey,
secretKey);
/**
* Initializing emr client object
*/
emrClient = new AmazonElasticMapReduceClient(credentials);
emrClient.setEndpoint(EmrUtils.ENDPOINT);
/**
* Specifying bootstrap actions
*/
ScriptBootstrapActionConfig scriptBootstrapConfig = new ScriptBootstrapActionConfig();
scriptBootstrapConfig.setPath("s3://bucket/bootstrapScript.sh");
BootstrapActionConfig bootstrapActions = new BootstrapActionConfig(
"Bootstrap Script", scriptBootstrapConfig);
RunJobFlowRequest jobFlowRequest = new RunJobFlowRequest()
.withName("Java SDK EMR cluster")
.withLogUri(EmrUtils.S3_LOG_URI)
.withAmiVersion(EmrUtils.AMI_VERSION)
.withBootstrapActions(bootstrapActions)
.withInstances(
new JobFlowInstancesConfig()
.withEc2KeyName(EmrUtils.EC2_KEY_PAIR)
.withHadoopVersion(EmrUtils.HADOOP_VERSION)
.withInstanceCount(1)
.withEc2SubnetId(EmrUtils.EC2_SUBNET_ID) // ******
.withKeepJobFlowAliveWhenNoSteps(true)
.withMasterInstanceType(EmrUtils.MASTER_INSTANCE_TYPE)
.withTerminationProtected(true)
.withSlaveInstanceType(EmrUtils.SLAVE_INSTANCE_TYPE));
RunJobFlowResult result = emrClient.runJobFlow(jobFlowRequest);
String jobFlowId = result.getJobFlowId();
System.out.println(jobFlowId);
} catch (Exception e) {
e.printStackTrace();
System.out.println("Shutting down cluster");
if (emrClient != null) {
emrClient.shutdown();
}
}
When I do the same thing using the EMR console, the cluster starts, bootstraps and successfully goes into the WAITING state. Is there any other way I can specify the subnet id to start a cluster. I suppose boto allows us to send additional parameters as a string. I found something similar in Java: .withAdditionalInfo(additionalInfo) which is a method of RunJobFlowRequest() and takes a JSON string as an argument. I don't however know the key that should be used for the ec2 subnet id in the JSON string.
(Using python boto is not an option for me, I've faced other showstopping issues with that and had to shift to AWS Java SDK)