here is some log from chrome://net-internals/#events
HTTP_TRANSACTION_SEND_REQUEST
t=18581 [st= 2] +HTTP_TRANSACTION_READ_HEADERS [dt=2744]
t=18581 [st= 2] HTTP_STREAM_PARSER_READ_HEADERS [dt=2744]
t=21325 [st=2746] HTTP_TRANSACTION_READ_RESPONSE_HEADERS
and the request pending almost 2700 ms. How can I solve this problem?
I have test that client,internet speed is fast,and ping my domain, also normal;
server(other user response quickly) and api(this api just print current time) also ok,
There may be several reasons of it
internet speed may be slow.
server configuration may be lower then required (RAM and processor).
too much conditions in api that you calling or there may complicated query in api(with joins and sub query).
check above possibilities in your code.
Related
I want my consumers to process large batches, so I aim to have the consumer listener "awake", say, on 1800mb of data or every 5min, whichever comes first.
Mine is a kafka-springboot application, the topic has 28 partitions, and this is the configuration I explicitly change:
Parameter
Value I set
Default Value
Why I set it this way
fetch.max.bytes
1801mb
50mb
fetch.min.bytes+1mb
fetch.min.bytes
1800mb
1b
desired batch size
fetch.max.wait.ms
5min
500ms
desired cadence
max.partition.fetch.bytes
1801mb
1mb
unbalanced partitions
request.timeout.ms
5min+1sec
30sec
fetch.max.wait.ms + 1sec
max.poll.records
10000
500
1500 found too low
max.poll.interval.ms
5min+1sec
5min
fetch.max.wait.ms + 1sec
Nevertheless, I produce ~2gb of data to the topic, and I see the consumer-listener (a Batch Listener) is called many times per second -- way more than desired rate.
I logged the serialized-size of the ConsumerRecords<?,?> argument, and found that it is never more than 55mb.
This hints that I was not able to set fetch.max.bytes above the default 50mb.
Any idea how I can troubleshoot this?
Edit:
I found this question: Kafka MSK - a configuration of high fetch.max.wait.ms and fetch.min.bytes is behaving unexpectedly
Is it really impossible as stated?
Finally found the cause.
There is a broker fetch.max.bytes setting, and it defaults to 55mb. I only changed the consumer preferences, unaware of the broker-side limit.
see also
The kafka KIP and the actual commit.
My server used to see APPARENT DEADLOCK in the logs. I have several servers running behind a load balancer, and the interesting thing is I see the DEADLOCK occur on all servers at the same time (does anyone know why it affects all servers)?. During this time period, MySQL queries that normally take 200ms take >60 seconds. Here's what logs looked like then:
com.mchange.v2.async.ThreadPoolAsynchronousRunner: com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector#58780f76
-- APPARENT DEADLOCK!!! Complete Status:
Managed Threads: 3
Active Threads: 3
Active Tasks:
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StatementCloseTask#25ff87d4 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0)
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StatementCloseTask#10ccf7ef (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1)
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StatementCloseTask#3305ec37 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2)
Pending Tasks:
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StatementCloseTask#39cc9e5a
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StmtAcquireTask#60d46f90
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StatementCloseTask#17509fea
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StatementCloseTask#b28bd63
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StatementCloseTask#56cbdc12
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StatementCloseTask#15a091b4
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StatementCloseTask#61ce325
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StmtAcquireTask#48119520
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StatementCloseTask#4032fb7c
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StmtAcquireTask#518eefff
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StmtAcquireTask#30ea3b20
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StmtAcquireTask#74960088
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StmtAcquireTask#23a8fc7d
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StmtAcquireTask#5ff0ee0
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StatementCloseTask#642d0644
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StatementCloseTask#207bc809
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StatementCloseTask#44d4936f
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StatementCloseTask#39a10d1b
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StatementCloseTask#3532334d
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StatementCloseTask#4bf79e62
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StatementCloseTask#2bd83398
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StmtAcquireTask#1a202a2d
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StatementCloseTask#3eacda7f
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StatementCloseTask#495f5746
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StmtAcquireTask#23f1f906
So I came to Stack Overflow and found this answer which suggested I set statementCacheNumDeferredCloseThreads to 1. I did this, and I see DEADLOCK less frequently and only on a few servers behind the load balancer instead of all.
The logs look a little different now, but during DEADLOCK period, queries still very long:
10 Oct 2018 06:33:32,037 [WARN] (Timer-0) com.mchange.v2.async.ThreadPoolAsynchronousRunner: com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector#4f39ad63 -- APPARENT DEADLOCK!!! Complete Status:
Managed Threads: 3
Active Threads: 3
Active Tasks:
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StatementCloseTask#34dee200 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2)
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StatementCloseTask#3727ee6b (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1)
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StatementCloseTask#4afb8b9 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0)
Pending Tasks:
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StmtAcquireTask#384a3b5b
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StmtAcquireTask#7bc700b0
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask#731bfd15
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask#a88e9bf
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask#63f18b56
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask#20f0c518
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StmtAcquireTask#caf7746
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask#41a7a27d
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask#2ee32a24
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StmtAcquireTask#81df2e5
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask#7f7fa1e7
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask#337503f
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask#34b2f877
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StmtAcquireTask#53dfbede
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StmtAcquireTask#512d5ddb
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask#68a25969
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask#4bf0754a
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask#65770ba4
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask#5e0f4154
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StmtAcquireTask#249c22ed
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask#6c8e5911
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StmtAcquireTask#3179550f
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask#15d8a795
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StmtAcquireTask#50966489
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask#4ecee95b
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StmtAcquireTask#35640ca0
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#6550f196
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#6816399
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#3fbcd623
Pool thread stack traces:
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
com.mysql.jdbc.PreparedStatement.realClose(PreparedStatement.java:2765)
com.mysql.jdbc.StatementImpl.close(StatementImpl.java:541)
com.mchange.v1.db.sql.StatementUtils.attemptClose(StatementUtils.java:41)
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StatementCloseTask.run(GooGooStatementCache.java:404)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]
com.mysql.jdbc.PreparedStatement.realClose(PreparedStatement.java:2765)
com.mysql.jdbc.StatementImpl.close(StatementImpl.java:541)
com.mchange.v1.db.sql.StatementUtils.attemptClose(StatementUtils.java:41)
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StatementCloseTask.run(GooGooStatementCache.java:404)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
com.mysql.jdbc.PreparedStatement.realClose(PreparedStatement.java:2765)
com.mysql.jdbc.StatementImpl.close(StatementImpl.java:541)
com.mchange.v1.db.sql.StatementUtils.attemptClose(StatementUtils.java:41)
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StatementCloseTask.run(GooGooStatementCache.java:404)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
Any idea how to fix this? I could try disable statement caching altogether but I'm concerned about the general performance hit. Some other relevant parameters:
minPoolSize = 30
maxPoolSize = 30
maxStatements = 100
unreturnedConnectionTimeout = 500
idleConnectionTestPeriod = 60
acquireIncrements = 3
C3p0 version = 0.9.1.2
Edit: I forgot to mention, during this improvement where I saw less deadlocks, I also increased maxStatements which could explain the improvement. However now I just see https://github.com/swaldman/c3p0/issues/53 which says version 0.9.2 introduces this new parameter statementCacheNumDeferredCloseThreads. My version is too old. I get no warnings/errors about this parameter not existing.
Maybe it's too late, but have you tried to increase the number of numHelperThreads?
I see such code segment in proposer_task(xcom_base.c)
if(threephase || ep->p->force_delivery){
push_msg_3p(ep->site, ep->p, ep->prepare_msg, ep->msgno, normal);
}else{
push_msg_2p(ep->site, ep->p);
}
the threepahse is int const threephase = 0 and force_delivery == 0 here
push_msg_eq is normal paxos include prepare, accept and learn phase
but push_msg_2p will skip prepare phase and directly send accept request
I want to know why, Thanks a lot.
If you look at the paper Paxos Made Simple page 10 paragraph 3 says:
A newly chosen leader executes phase 1 for infinitely many instances
of the consensus algorithm [...]
Then paragraph 4:
Since failure of the leader and election of a new one should be rare
events, the effective cost of executing a state machine command—that
is, of achieving consensus on the command/value—is the cost of
executing only phase 2 of the consensus algorithm. It can be shown
that phase 2 of the Paxos consensus algorithm has the minimum possible
cost of any algorithm for reaching agreement in the presence of faults.
Hence, the Paxos algorithm is essentially optimal.
This is saying that a leader only issues a prepare during a leader failover. After that it streams accept messages. It then has "optimal messaging" in that the leader only needs one round trip to know a value is chosen (the accept message and its acknowledgment).
In a three node cluster, a leader self-accepts instantaneously, then only needs one accept acknowledgment from a second node to have a majority. It then knows the value is chosen without having to await the response from the 3rd node (which could be down). That is as efficient as you can get. The value is known to be accepted at a second node with strong consistency.
Given that is how paxos works to get maximum efficiency we should expect that mysql xcom has a mode that skips the prepare message phase in steady state.
You can read more about the Paxos Made Simple techniques on my blog here.
You might be interested to know about the latest developments of Paxos where you don't need a majority response for accept messages in the cluster using FPaxos and tricks like the even nodes optimization.
In Jmeter v2.13, is there a way to capture Throughput via non-GUI/Command Line mode?
I have the jmeter.properties file configured to output via the Summariser and I'm also outputting another [more detailed] .csv results file.
call ..\..\binaries\apache-jmeter-2.13\bin\jmeter -n -t "API Performance.jmx" -l "performanceDetailedResults.csv"
The performanceDetailedResults.csv file provides:
timeStamp
elapsed time
responseCode
responseMessage
threadName
success
failureMessage
bytes sent
grpThreads
allThreads
Latency
However, no amount of tweaking the .properties file or the test itself seems to provide Throuput results like I get via the GUI's Summary Report's Save Table Data button.
All articles, postings, and blogs seem to indicate it wasn't possible without manual manipulation in a spreadsheet. But I'm hoping someone out there has figured out a way to do this with no, or minimal, manual manipulation as the client doesn't want to have to manually calculate the Throughput value each time.
It is calculated by JMeter Listeners so it isn't something you can enable via properties files. Same applies to other metrics which are calculated like:
Average response time
50, 90, 95, and 99 percentiles
Standard Deviation
Basically throughput is calculated as simple as dividing total number of requests by elapsed time.
Throughput is calculated as requests/unit of time. The time is calculated from the start of the first sample to the end of the last sample. This includes any intervals between samples, as it is supposed to represent the load on the server.
The formula is: Throughput = (number of requests) / (total time)
Hopefully it won't be too hard for you.
References:
Glossary #1
Glossary #2
Did you take a look at JMeter-Plugins?
This tool can generate aggregate report through commandline.
I need the figure out how to manage my retries in Nservicebus.
If there is any exception in my flow, It should retry 10 times every 10 seconds. But when I search in Nservicebus' website (http://docs.particular.net/nservicebus/errors/automatic-retries), there are 2 different retry mechanisms which are First Level Retry(FLR) and Second Level Retry (SLR).
FLR is for transient errors. When you got exception, It will try instantly according to your MaxRetries parameter. This parameter should be 1 for me.
SLR is for errors that persist after FLR, where a small delay is needed between retries. There is a config parameter called "TimeIncrease" defines a delay time between tries. However, Nservicebus do these retries increasingly delay time. When you set this parameter to 10 second. It will try 10.seconds, 30.seconds, 60.seconds and so on.
What do you suggest to me to provide my first request to try every 10 seconds with or without these mechanisms?
I found my answer;
The reply of Particular Software's community(John Simon), You need to apply a custom retry policy, have a look at http://docs.particular.net/nservicebus/errors/automatic-retries#second-level-retries-custom-retry-policy-simple-policy for an example.