As a project we have setup a jmeter cluster with a master and 5 salve nodes.
We also configured influxDb for storing the metrics and grafana for visualization.
we have connection from jmeter to influxdb and are able to visualize the data.
The influxdb confguration is done on the master and if we run standalone on master, we are able to see the live data with all the detailed visualization in Grafana.
Issue : When we run the test in distributed mode, we see the result on grafana, but it looks all broken, looks all the data is not getting into influxdb.
How to get all the details of the slave execution to influxdb?
Are there any specific configuration to get all the details from jmeter to influxdb.
Thanks for your help.
Make sure that your slave instances are able to connect to InfluxDB and store the metrics there, i.e. verify connectivity to InfluxDB TCP or HTTP port using telnet client or equivalent, it might be the case the connections are blocked by the firewall
Check jmeter-server.log file on slave machines for any suspicious entries. You might want to increase JMeter logs verbosity for the Backend Listener by adding the next line to log4j2.xml file:
<Logger name="org.apache.jmeter.visualizers.backend.influxdb" level="debug"/>
in case of success you should see the lines like:
HttpMetricsSender: Success, number of metrics written: x
...
InfluxdbBackendListenerClient: Sending last metrics
If you want to see separate results for Samplers originating from different slaves consider adding __machineName() or __machineIP() function to Samplers labels
Related
I need to validate a workload on a DB used to answer to http api.
In this context, on production, there are a lot of connections opened / closed. For a connection, there are only 2 or 3 small queries launched.. So connection 'activity' (open/close) has to be taken into account in our application.
I need to 'bench' / test the DB without the application stack, so I'd like JMETER to query directly the database like the web service would do..
When using / configuring odbc connection pool through "jdbc connection configuration", I only see the way to define a large pool of connection that will be used, after, to launch queries. That mean... the connections stay alive after playing ThreadGroup scenario, and are reused. In real application, for a scenario, this would make a new connection, and would close this one at the end.
Is there a way to do it (make a new connection for every ThreadGroup run) in JMETER with JDBC 'components' ?
as a workarround, I created a small script and asked jmeter to run it... but it's far more heavier for the server to do it (launch a new process each time to execute the (php) script.. and I couldn't load the server enough by doing it, to reproduce the workload.
JMeter is actually calling Connection.close() function after executing the statement, under the hood the connection is being returned to the pool and it waits for the next thread which requires the connection.
If your application behaviour is the same you don't need to worry about anything. If it's different - you won't get such precise control with the JDBC Connection Configuration and JDBC Request sampler.
If you want to create and destroy connections manually you will have to switch to JSR232 Sampler and implement connection and query logic in Groovy, see Working with a relational database Groovy user manual chapter for more details, code examples, etc.
MarkLogic setup is as follows
3 hosts
Data confniguration
- 1 master forest on each host
- 1 replica for each host on different host
We have MarkLogic cluster (3 hosts) with failover) deployed on Azure VMs
We are using MarkLogic ContentPump (MLCP) to ingest data into MarkLogic
This is what we have implemented
Installed Java on 1st host
Copied MLCP tool
Ingested data by providing 1st server as host parameter
Now we got batch of xmls to update back to MarkLogic
With failover implementation, due to some reason 1st host is not available, so when i tried to ingest data thru 2nd host, i started getting error that record was ingested in different host, so update can't happen from here.
So i would like to know the best practices to be followed for ingestion process
To enable the system to reliably failover, you will also need to setup replicas for the Security, App Services & any other system database you may be using as part of your architecture.
The reason you are unable to connect to the other hosts is that the Security database is on host 1, so you are unable to authenticate. Once that is configured for failover, you should no longer run into those issues.
The documentation covers that setup here:
https://docs.marklogic.com/guide/cluster/config-both-failover#id_57935
I am getting the following error:
"ERROR WorkerSourceTask(id=test-mysql-dbc-source-0) Failed to flush, timed out while waiting for producer to flush outstanding N messages.
ERROR Failed to commit offsets. (org.apache.kafka.connect.runtime.WorkerSourceTask:XXX)"
The Setup:
I have an EC2 instance in AWS (t2.medium - 2 cores 4GB RAM) which serves as Kafka Server. Another EC2 instance has a sandbox MySQL database and Kafka Connect with Confluent JDBC Source Connector. Python script inserts a couple of rows in a table randomly, to simulate some activity. On my laptop I opened Kafka Console Consumer to read the messages from Kafka Server.
Ports 22, 3306, 9092, 2888 are open to all IP addresses on both EC2 instances.
Below are config files I used for Kafka Connect Source
Config files:
connect-standalone.properties
bootstrap.servers=FIRST_EC2_PUBLIC_IP:9092
key.converter.schemas.enable=true
value.converter.schemas.enable=true
offset.storage.file.filename=/tmp/connect.offsets
acks=1
compression.type=lz4
plugin.path=/usr/share/java
jdbc_source.properties
name=test-mysql-jdbc-source
connector.class=io.confluent.connect.jdbc.JdbcSourceConnector
tasks.max=1
connection.url=jdbc:mysql://localhost:3306/DB_NAME
connection.user=root
connection.password=DB_PASSWORD
table.whitelist=TEST_TABLE
mode=timestamp+incrementing
incrementing.column.name=ID_RECORD
timestamp.column.name=TIMESTMP
topic.prefix=mysql-source-
acks=1
compression.type=lz4
I tried to manipulate with various settings and options. Mostly I tried to play around with offset.flush.timeout.ms and buffer.memory options as advised in this link.
The Behavior of Producer:
So basically, after starting a producer on my second EC2 instance, I can see the messages on my laptop in kafka console consumer, so it works. I can see new records as they appear for some time. However, very often when a new row in a table is created, producer just does not push it to kafka server (first EC2 instance) throwing above mentioned error for about 5 to 20 minutes. The number of outstanding messages doesn't get big. Most of the time it is 2-6 messages. After throwing an error for 5-20 minutes it finally sends the data and the console consumer consumes the data and work fine for some time and after that an error appears again.
If I manually stop the producer and start it again, the outstanding messages flush instantly and can be seen in a console consumer on my laptop.
Could you please point me, where the problem can be, and what can cause such a behavior.
I try to configure RabbitMQ cluster in a cloud using config file.
My procedure is this:
Get list of instances I want to cluster with (via cloud API, before cluster startup).
Modify config file like that:
cluster_formation.peer_discovery_backend = rabbit_peer_discovery_classic_config
cluster_formation.classic_config.nodes.1 = rabbit#host1.my.long.domain.name
cluster_formation.classic_config.nodes.2 = rabbit#host2.my.long.domain.name
...
Run rabbitmq-server
I expect all nodes to cluster but instead there might be 2+ independent clusters. How do I solve this issue?
UPDATE:
I found out that when I run rabbitmqctl join_cluster rabbit#host.in.existing.cluster on node that is in some cluster already, this node leaves his previous cluster (I expected this clusters to merge). That might be root of the problem.
UPDATE 2:
I have 4 instances. 3 run bare rabbitmq-servers, 1 is configured to join other 3. When started, it joins with the last instance in its config, 2 others show no activity in threir logs. Happens on classic config and erlang config both.
When you initially start up your cluster, there is no means to resolve race condition. Using peer discovery backends will not help with this issue (tested on etcd).
What actually resolved this issue is not deploying instances simultaneously. When started one by one everything is fine and you get one stable cluster which can handle scaling without failure.
I am using Apache Spark to run machine learning algorithms and other big data tasks. Previously, I was using spark cluster standalone mode running spark master and worker on the same machine. Now, I added multiple worker machines and due to a tight firewall, I have to edit the random port of worker. Can anyone help how to change random spark ports and tell me exactly what configuration file needs to be edited? I read the spark documentation and it says spark-defaults.conf should be configured but I don't know how I can configure this file for particularly changing random ports of spark.
Update for Spark 2.x
Some libraries have been rewritten from scratch and many legacy *.port properties are now obsolete (cf. SPARK-10997 / SPARK-20605 / SPARK-12588 / SPARK-17678 / etc)
For Spark 2.1, for instance, the port ranges on which the driver will listen for executor traffic are
between spark.driver.port and spark.driver.port+spark.port.maxRetries
between spark.driver.blockManager.port and spark.driver.blockManager.port+spark.port.maxRetries
And the port range on which the executors will listen for driver traffic and/or other executors traffic is
between spark.blockManager.port and spark.blockManager.port+spark.port.maxRetries
The "maxRetries" property allows for running several Spark jobs in parallel; if the base port is already used, then the new job will try the next one, etc, unless the whole range is already used.
Source:
https://spark.apache.org/docs/2.1.1/configuration.html#networking
https://spark.apache.org/docs/2.1.1/security.html under "Configuring ports"
check here https://spark.apache.org/docs/latest/configuration.html#networking
In the "Networking" section, you can see some of the ports are by default random. You can set them to your choice like below:
val conf = new SparkConf()
.setMaster(master)
.setAppName("namexxx")
.set("spark.driver.port", "51810")
.set("spark.fileserver.port", "51811")
.set("spark.broadcast.port", "51812")
.set("spark.replClassServer.port", "51813")
.set("spark.blockManager.port", "51814")
.set("spark.executor.port", "51815")