Web spider speed bound by MySQL config? - mysql

Question

I recently moved my web spider to a paid cloud type hosting situation after reaching the limitations of my office network on Comcast Business. I am experiencing some strange limitation that hopefully someone can shed some light on.
I have 10 small instances running - each one has 1 GB of ram, and 1 CPU - with SSd storage.
Each instance is configured to visit basically 1 million websites per day - and each instance when run by itself can easily achieve this.
I have another instance that is handling a simple MySQL database for keeping track - this db uses 8 GB of ram, 4 cores, and 90 GB of SSD.
If I run 4 instances I can hit 4 million per day, if I run 10 instances i`m still only hitting 4 million per day - something is choking in this configuration.
MySQL runs at about 360% of CPU (4 cores) most of the time and about 70% of memory - typical io is about 4 MB/s.
The database writes consist of 10 tables and not every visit will result in a write to every column - about 50% of visits result in 2-10 table writes. The only table that always updates is the "visited" table with the last visit date/time.
Here are some excerpts from my config:
my.cnf:
explicit_defaults_for_timestamp
connect_timeout = 60
sync_binlog = 0
innodb_buffer_pool_size = 5G
innodb_file_format=Barracuda
innodb_log_file_size = 10G
innodb_file_per_table=1
innodb_log_buffer_size=4M
innodb_flush_log_at_trx_commit=0
innodb_thread_concurrency=10
#transaction-isolation=READ-COMMITTED
max_connections = 2500
innodb_buffer_pool_instances = 5
innodb_io_capacity = 30000
innodb_read_io_threads = 10000
innodb_write_io_threads = 10000
innodb_doublewrite = 0
innodb_open_files = 10000
innodb_support_xa=0
innodb_flush_method = O_DIRECT
max_allowed_packet = 32M
thread_stack = 1M
sort_buffer_size = 256K
table_open_cache = 2000
thread_cache_size = 5000
Here is a typical table - with indexes & so forth - I believe these are fairly dialed since I can query/sort/etc very quickly.
CREATE TABLE `websites` (
`wid` bigint(20) NOT NULL AUTO_INCREMENT,
`host` varchar(100) NOT NULL,
`status` int(3) NOT NULL DEFAULT '0',
`total_time` int(15) NOT NULL DEFAULT '0',
`total_data` int(15) NOT NULL DEFAULT '0',
`hash` int(15) NOT NULL DEFAULT '0',
`machine` int(2) NOT NULL DEFAULT '0',
`ipv4` int(4) unsigned DEFAULT NULL,
`ipv6` binary(16) DEFAULT NULL,
PRIMARY KEY (`wid`),
UNIQUE KEY `host` (`host`),
KEY `status` (`status`),
KEY `total_time` (`total_time`),
KEY `total_data` (`total_data`),
KEY `machine` (`machine`),
KEY `ipv4` (`ipv4`,`ipv6`),
KEY `hash` (`hash`) USING BTREE
) ENGINE=InnoDB AUTO_INCREMENT=3741662307 DEFAULT CHARSET=utf8 ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=4
None of the SELECT queries are ever the same.
SHOW GLOBAL STATUS;
Aborted_clients
6728
Aborted_connects
135
Binlog_cache_disk_use
0
Binlog_cache_use
0
Binlog_stmt_cache_disk_use
0
Binlog_stmt_cache_use
0
Bytes_received
42547122115
Bytes_sent
36810202741
Com_admin_commands
0
Com_assign_to_keycache
0
Com_alter_db
0
Com_alter_db_upgrade
0
Com_alter_event
0
Com_alter_function
0
Com_alter_procedure
0
Com_alter_server
0
Com_alter_table
0
Com_alter_tablespace
0
Com_alter_user
0
Com_analyze
0
Com_begin
0
Com_binlog
0
Com_call_procedure
0
Com_change_db
167
Com_change_master
0
Com_change_repl_filter
0
Com_check
0
Com_checksum
0
Com_commit
0
Com_create_db
0
Com_create_event
0
Com_create_function
0
Com_create_index
0
Com_create_procedure
0
Com_create_server
0
Com_create_table
0
Com_create_trigger
0
Com_create_udf
0
Com_create_user
0
Com_create_view
0
Com_dealloc_sql
0
Com_delete
0
Com_delete_multi
0
Com_do
0
Com_drop_db
0
Com_drop_event
0
Com_drop_function
0
Com_drop_index
0
Com_drop_procedure
0
Com_drop_server
0
Com_drop_table
0
Com_drop_trigger
0
Com_drop_user
0
Com_drop_view
0
Com_empty_query
0
Com_execute_sql
0
Com_explain_other
0
Com_flush
0
Com_get_diagnostics
0
Com_grant
0
Com_ha_close
0
Com_ha_open
0
Com_ha_read
0
Com_help
0
Com_insert
261004903
Com_insert_select
0
Com_install_plugin
0
Com_kill
0
Com_load
0
Com_lock_tables
0
Com_optimize
0
Com_preload_keys
0
Com_prepare_sql
0
Com_purge
0
Com_purge_before_date
0
Com_release_savepoint
0
Com_rename_table
0
Com_rename_user
0
Com_repair
0
Com_replace
14
Com_replace_select
0
Com_reset
0
Com_resignal
0
Com_revoke
0
Com_revoke_all
0
Com_rollback
0
Com_rollback_to_savepoint
0
Com_savepoint
0
Com_select
225216121
Com_set_option
12406
Com_signal
0
Com_show_binlog_events
0
Com_show_binlogs
12
Com_show_charsets
0
Com_show_collations
0
Com_show_create_db
0
Com_show_create_event
0
Com_show_create_func
0
Com_show_create_proc
0
Com_show_create_table
47
Variable_name
Value
Com_show_create_trigger
0
Com_show_databases
0
Com_show_engine_logs
0
Com_show_engine_mutex
0
Com_show_engine_status
0
Com_show_events
0
Com_show_errors
0
Com_show_fields
585
Com_show_function_code
0
Com_show_function_status
0
Com_show_grants
4
Com_show_keys
46
Com_show_master_status
9
Com_show_open_tables
0
Com_show_plugins
0
Com_show_privileges
0
Com_show_procedure_code
0
Com_show_procedure_status
0
Com_show_processlist
26
Com_show_profile
0
Com_show_profiles
0
Com_show_relaylog_events
0
Com_show_slave_hosts
0
Com_show_slave_status
9
Com_show_status
2
Com_show_storage_engines
0
Com_show_table_status
0
Com_show_tables
32
Com_show_triggers
0
Com_show_variables
6462
Com_show_warnings
0
Com_show_create_user
0
Com_slave_start
0
Com_slave_stop
0
Com_group_replication_start
0
Com_group_replication_stop
0
Com_stmt_execute
0
Com_stmt_close
0
Com_stmt_fetch
0
Com_stmt_prepare
0
Com_stmt_reset
0
Com_stmt_send_long_data
0
Com_truncate
0
Com_uninstall_plugin
0
Com_unlock_tables
0
Com_update
3768683
Com_update_multi
0
Com_xa_commit
0
Com_xa_end
0
Com_xa_prepare
0
Com_xa_recover
0
Com_xa_rollback
0
Com_xa_start
0
Com_stmt_reprepare
0
Connection_errors_accept
0
Connection_errors_internal
0
Connection_errors_max_connections
0
Connection_errors_peer_address
0
Connection_errors_select
0
Connection_errors_tcpwrap
0
Connections
3781613
Created_tmp_disk_tables
900
Created_tmp_files
153
Created_tmp_tables
1714
Delayed_errors
0
Delayed_insert_threads
0
Delayed_writes
0
Flush_commands
1
Handler_commit
489978593
Handler_delete
0
Handler_discover
0
Handler_external_lock
979991125
Handler_mrr_init
0
Handler_prepare
0
Handler_read_first
1022
Handler_read_key
340529954
Handler_read_last
14
Handler_read_next
1325647372
Handler_read_prev
240
Handler_read_rnd
111545828
Handler_read_rnd_next
3254347
Handler_rollback
9905
Handler_savepoint
0
Handler_savepoint_rollback
0
Handler_update
3768685
Handler_write
261011792
Innodb_buffer_pool_dump_status
not started
Innodb_buffer_pool_load_status
Buffer pool(s) load completed at 150906 5:57:38
Innodb_buffer_pool_resize_status
not started
Innodb_buffer_pool_pages_data
909831
Innodb_buffer_pool_bytes_data
5310021632
Innodb_buffer_pool_pages_dirty
137841
Innodb_buffer_pool_bytes_dirty
586342400
Innodb_buffer_pool_pages_flushed
156711220
Innodb_buffer_pool_pages_free
2828
Innodb_buffer_pool_pages_misc
18446744073708966597
Innodb_buffer_pool_pages_total
327640
Innodb_buffer_pool_read_ahead_rnd
0
Innodb_buffer_pool_read_ahead
115535
Innodb_buffer_pool_read_ahead_evicted
0
Variable_name
Value
Innodb_buffer_pool_read_requests
17132988745
Innodb_buffer_pool_reads
43009838
Innodb_buffer_pool_wait_free
0
Innodb_buffer_pool_write_requests
3418426697
Innodb_data_fsyncs
748775
Innodb_data_pending_fsyncs
0
Innodb_data_pending_reads
0
Innodb_data_pending_writes
0
Innodb_data_read
182409891840
Innodb_data_reads
44358403
Innodb_data_writes
17867174
Innodb_data_written
224979945984
Innodb_dblwr_pages_written
0
Innodb_dblwr_writes
0
Innodb_log_waits
0
Innodb_log_write_requests
343630408
Innodb_log_writes
133630
Innodb_os_log_fsyncs
92883
Innodb_os_log_pending_fsyncs
0
Innodb_os_log_pending_writes
0
Innodb_os_log_written
149354686976
Innodb_page_size
16384
Innodb_pages_created
1520762
Innodb_pages_read
44358411
Innodb_pages_written
17719344
Innodb_row_lock_current_waits
0
Innodb_row_lock_time
125773
Innodb_row_lock_time_avg
41
Innodb_row_lock_time_max
51019
Innodb_row_lock_waits
3015
Innodb_rows_deleted
0
Innodb_rows_inserted
47941507
Innodb_rows_read
1442492244
Innodb_rows_updated
3768685
Innodb_num_open_files
42
Innodb_truncated_status_writes
0
Innodb_available_undo_logs
128
Key_blocks_not_flushed
0
Key_blocks_unused
6696
Key_blocks_used
2
Key_read_requests
376
Key_reads
6
Key_write_requests
0
Key_writes
0
Locked_connects
0
Max_execution_time_exceeded
0
Max_execution_time_set
0
Max_execution_time_set_failed
0
Max_used_connections
795
Max_used_connections_time
2015-09-06 21:13:06
Not_flushed_delayed_rows
0
Ongoing_anonymous_transaction_count
0
Open_files
32
Open_streams
0
Open_table_definitions
132
Open_tables
2000
Opened_files
384
Opened_table_definitions
132
Opened_tables
26470
Performance_schema_accounts_lost
0
Performance_schema_cond_classes_lost
0
Performance_schema_cond_instances_lost
0
Performance_schema_digest_lost
0
Performance_schema_file_classes_lost
0
Performance_schema_file_handles_lost
0
Performance_schema_file_instances_lost
0
Performance_schema_hosts_lost
0
Performance_schema_index_stat_lost
0
Performance_schema_locker_lost
0
Performance_schema_memory_classes_lost
0
Performance_schema_metadata_lock_lost
0
Performance_schema_mutex_classes_lost
0
Performance_schema_mutex_instances_lost
0
Performance_schema_nested_statement_lost
0
Performance_schema_prepared_statements_lost
0
Performance_schema_program_lost
0
Performance_schema_rwlock_classes_lost
0
Performance_schema_rwlock_instances_lost
0
Performance_schema_session_connect_attrs_lost
0
Performance_schema_socket_classes_lost
0
Performance_schema_socket_instances_lost
0
Performance_schema_stage_classes_lost
0
Performance_schema_statement_classes_lost
0
Performance_schema_table_handles_lost
0
Performance_schema_table_instances_lost
0
Performance_schema_table_lock_stat_lost
0
Performance_schema_thread_classes_lost
0
Performance_schema_thread_instances_lost
0
Performance_schema_users_lost
0
Prepared_stmt_count
0
Qcache_free_blocks
1
Qcache_free_memory
1031832
Qcache_hits
0
Qcache_inserts
0
Qcache_lowmem_prunes
0
Qcache_not_cached
225057348
Qcache_queries_in_cache
0
Qcache_total_blocks
1
Queries
493783842
Questions
493783908
Variable_name
Value
Select_full_join
56
Select_full_range_join
0
Select_range
158
Select_range_check
0
Select_scan
8023
Slave_open_temp_tables
0
Slow_launch_threads
0
Slow_queries
0
Sort_merge_passes
3948
Sort_range
49
Sort_rows
490845
Sort_scan
129
Ssl_accept_renegotiates
0
Ssl_accepts
0
Ssl_callback_cache_hits
0
Ssl_cipher
Ssl_cipher_list
Ssl_client_connects
0
Ssl_connect_renegotiates
0
Ssl_ctx_verify_depth
0
Ssl_ctx_verify_mode
0
Ssl_default_timeout
0
Ssl_finished_accepts
0
Ssl_finished_connects
0
Ssl_server_not_after
Aug 29 23:50:49 2025 GMT
Ssl_server_not_before
Sep 1 23:50:49 2015 GMT
Ssl_session_cache_hits
0
Ssl_session_cache_misses
0
Ssl_session_cache_mode
Unknown
Ssl_session_cache_overflows
0
Ssl_session_cache_size
0
Ssl_session_cache_timeouts
0
Ssl_sessions_reused
0
Ssl_used_session_cache_entries
0
Ssl_verify_depth
0
Ssl_verify_mode
0
Ssl_version
Table_locks_immediate
6801
Table_locks_waited
0
Table_open_cache_hits
489970063
Table_open_cache_misses
26470
Table_open_cache_overflows
24463
Tc_log_max_pages_used
0
Tc_log_page_size
0
Tc_log_page_waits
0
Threads_cached
423
Threads_connected
370
Threads_created
795
Threads_running
310
Uptime
57104
Uptime_since_flush_status
57104
iostat
Linux 3.16.0-4-amd64 (vultr.guest) 09/06/2015 _x86_64_ (4 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
46.63 0.00 8.83 1.86 0.31 42.37
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
vda 1149.87 4598.30 3737.19 403966815 328316761
Running debian Jessie, MySQL 5.7, and using services from vultr.com

Answer 1

Alright so after a night of tweaking & watching this fantastic tool - innotop - I zeroed in on the website table being updated by INSERT IGNORE INTO websites host=somehost.com - this is a fairly uncomplex query that runs very quickly. However in my case there are half a billion domains in the database, and some very common domains such as w3.org show up all the time, and are constantly re-sent to the mothership for no reason. My solution was to create a smaller host only database on each of the nodes, where all "seen" hosts are stored - if the host has been seen, it will never be sent to the mothership again.
Now, normally you'd run a simple perl hash to track this, but since my script uses Parallel::Forkmanager for children, they don't really talk to each other.
Currently my throughput has doubled, and after filling out that small hosts table on each node with the entire domain list it will be even faster. I may do this with other tables that have the same INSERT IGNORE queries, and further speed things up - but for now i`m hitting my target per day and will settle for that...