Rails 4: Unexpected slowdowns in view rendering and ActiveRecord - mysql

Question

I am developing application on Rails and have troubles with performance. Below I have described and provided all technologies, configurations and methods I used. I tried many methods to solve this problem but nothing helped. This is also described below. Can anyone see what the problem is? Thank you!
The problem
Let's open site several times and see log:
INFO -- : Completed 200 OK in 171ms (Views: 149.3ms | ActiveRecord: 7.3ms)
INFO -- : Completed 200 OK in 217ms (Views: 183.7ms | ActiveRecord: 8.9ms)
INFO -- : Completed 200 OK in 221ms (Views: 188.2ms | ActiveRecord: 11.7ms)
INFO -- : Completed 200 OK in 165ms (Views: 143.3ms | ActiveRecord: 7.1ms)
Fine.
Now let's generate load with wrk -t10 -c10 -d10s http://example.com and see log again:
INFO -- : Completed 200 OK in 178ms (Views: 157.6ms | ActiveRecord: 8.1ms)
INFO -- : Completed 200 OK in 270ms (Views: 241.6ms | ActiveRecord: 8.8ms)
INFO -- : Completed 200 OK in 505ms (Views: 460.7ms | ActiveRecord: 12.0ms)
INFO -- : Completed 200 OK in 501ms (Views: 350.8ms | ActiveRecord: 136.0ms)
INFO -- : Completed 200 OK in 777ms (Views: 468.1ms | ActiveRecord: 269.1ms)
INFO -- : Completed 200 OK in 936ms (Views: 624.7ms | ActiveRecord: 285.1ms)
...
INFO -- : Completed 200 OK in 881ms (Views: 617.4ms | ActiveRecord: 251.6ms)
...
INFO -- : Completed 200 OK in 3289ms (Views: 2614.2ms | ActiveRecord: 326.4ms)
INFO -- : Completed 200 OK in 3369ms (Views: 2624.0ms | ActiveRecord: 409.4ms)
INFO -- : Completed 200 OK in 3203ms (Views: 2369.2ms | ActiveRecord: 382.1ms)
...
INFO -- : Completed 200 OK in 431ms (Views: 281.1ms | ActiveRecord: 126.3ms)
...
What are this terrible slowdowns?
wrk -t10 -c10 -d10s http://example.com
Running 10s test # http://example.com
10 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 1.26s 96.38ms 1.39s 75.00%
Req/Sec 0.00 0.00 0.00 100.00%
36 requests in 10.09s, 0.97MB read
Socket errors: connect 0, read 0, write 0, timeout 32
Requests/sec: 3.57
Transfer/sec: 98.62KB
Rails
I am using latest Rails version - 4.2.4.
bundle list output - github gist
application.rb - github gist
production.rb - github gist
database.yml - github gist
I am using acts_as_paranoid for all my models. (WHERE deleted_at IS NULL)
All N+1 queries are eliminated (analyzed them manually and with bullet gem).
All tables are InnoDB and have a lot of indices. 99% of tables have 50 or less rows, the rest 1% - 200 or less. That's very low amount of data.
Page I am benchmarking requires around 30-35 SQL queries and around 30 partials.
Queries using...
- COUNT - 0
- GROUP BY - 1
- ORDER BY - 10
- LIMIT - 10
- JOIN - 1 (joins one table)
- Use of MAX, MIN, CONCAT and other functions - NO
My templates are written in Slim.
Server configuration
CPU - Intel one core Sandy Bridge CPU
RAM - 1GM
Disk Type - SSD
Operating System
Ubuntu 14.04 LTS 64bit
Swap 1GB. Optimized swap settings:
cat /proc/sys/vm/swappiness => 10
cat /proc/sys/vm/vfs_cache_pressure => 50
Database
MySQL 5.5. Tried to optimize with mysqltuner and tuning-primer.
Current configuration: github gist
Latest mysqltuner report: perl mysqltuner.pl --buffers --dbstat --outputfile result_mysqltuner.txt github gist
Latest tuning-primer report: ./tuning-primer.sh github gist
I have notices the warning Temporary tables created on disk. That's because I have TEXT columns in MySQL. Googling I found that mounting MySQL temporary directory in memory is good idea. So I did it: tmpdir /var/mysqltmp. Followed this guide.
mytop output:
Queries: 37.4k qps: 0 Slow: 1.9k Se/In/Up/De(%): 91/00/00/00
Sorts: 0 qps now: 1 Slow qps: 0.0 Threads: 4 ( 1/ 2) 00/00/00/00
Cache Hits: 29.0k Hits/s: 0.4 Hits now: 0.0 Ratio: 85.5%
Ratio now: 0.0%
Key Efficiency: 99.4% Bps in/out: 75.8/ 1.9k Now in/out: 22.6/ 2.1k
Id User Host/IP DB Time Cmd State Query
-- ---- ------- -- ---- --- ----- ----------
1444 root localhost site 73737 Sleep
2337 root localhost site 658 Sleep
2340 root localhost site 322 Sleep
2342 root localhost 0 Query show full processlist
Ruby interpreter
Rbenv: rbenv -v => rbenv 0.4.0-154-g9e664b5
Ruby: ruby -v => ruby 2.2.3p173 (2015-08-18 revision 51636) [x86_64-linux]
My ruby is compiled with jemalloc (better memory allocation):
RbConfig::CONFIG['LIBS'] => "-lpthread -ljemalloc -ldl -lcrypt -lm "
Followed this guide.
I have tried both ruby with default and custom memory allocation. Ruby with jemalloc seems to be faster.
Web server
nginx -v => nginx version: nginx/1.8.0
Current nginx.conf (simplified): github gist
Current nginx site conf (simplified): github gist
Application server
I have tried both Passenger and Puma. But without result.
Phusion Passenger
passenger -v => Phusion Passenger version 5.0.18
Configuration: github gist
Puma
bundle exec puma -v => puma version 2.13.4
Also I have tried Puma server in both cluster (1 worker) and thread models.
Here is latest config for Puma in thread mode: github gist