Asynchronous operations with Hangfire - mysql

I've setup a simple .netcore 3.1 web application, based on Ubuntu 16.04 Server with Mysql 8.0.25 and Hangfire 1.7.30 for scheduled activities.
I'm actually struggling in understanding how scheduling activities should be made. In the actual version of the SW, I've created up to 8 asynchronous scheduled tasks managed by Hangfire, all of them structured as follow:
public static async Task ScheduledTaskAsync_UpdateTransferList()
{
await TaskScheduling.UpdateTransferList(SavedAppServices);
}
All these scheduling activities query the mysql DB, that's why I've made them asynchronous.
One of these 8 services is quite frequent (Cron.Minutely()) and takes few seconds to be executed.
I'm wondering if I'm doing something wrong, because I see in the app log many consecutive messages like:
dbug: Hangfire.Server.RecurringJobScheduler[0]
1000 recurring job(s) processed by scheduler.
sometimes for a total of 9000-10000 recurring jobs processed.
Furthermore, after moving from Mysql 5.7.3 to 8.0.25, I've noticed a sharp increase of mysql binlogs (up to 6Gb generated in two days), all of them because of multiple logs like this (generated at the same second):
# at 9683673
#220617 5:25:45 server id 1 end_log_pos 9683704 CRC32 0x08936f4a Xid = 25774476
COMMIT/*!*/;
# at 9683704
#220617 5:25:45 server id 1 end_log_pos 9683783 CRC32 0xdd8b5fe2 Anonymous_GTID last_committed=26277 sequence_number=26278 rbr_only=yes original_committed_timestamp=1655443545028260 immediate_commit_timestamp=1655443545028260 transaction_length=332
/*!50718 SET TRANSACTION ISOLATION LEVEL READ COMMITTED*//*!*/;
# original_commit_timestamp=1655443545028260 (2022-06-17 05:25:45.028260 UTC)
# immediate_commit_timestamp=1655443545028260 (2022-06-17 05:25:45.028260 UTC)
/*!80001 SET ##session.original_commit_timestamp=1655443545028260*//*!*/;
/*!80014 SET ##session.original_server_version=80025*//*!*/;
/*!80014 SET ##session.immediate_server_version=80025*//*!*/;
SET ##SESSION.GTID_NEXT= 'ANONYMOUS'/*!*/;
# at 9683783
#220617 5:25:45 server id 1 end_log_pos 9683862 CRC32 0x8b83c028 Query thread_id=110 exec_time=0 error_code=0
SET TIMESTAMP=1655443545/*!*/;
BEGIN
/*!*/;
# at 9683862
#220617 5:25:45 server id 1 end_log_pos 9683940 CRC32 0x29c7f191 Table_map: `Hangfire`.`Hangfire.DistributedLock` mapped to number 92
# at 9683940
#220617 5:25:45 server id 1 end_log_pos 9684005 CRC32 0x6f8c7b99 Delete_rows: table id 92 flags: STMT_END_F
To be noted that the Hangfire Dashboard panel signals no issues and I know that I could simply deactivate binlog generation, but I would like to understand if I'm doing something wrong with Hangfire/Mysql and how I can make everything work without a so huge waste of diskspace due to binlog generation.

Related

geth does not persist trie node data from memory to disk on ungraceful system restart

Issue: geth 1.8.22 starts mining from one of the first blocks instead of the last one on system reboot.
What we have
We have 3 synced private geth nodes using PoA(clique).
What happened
One day(a week ago) we had issues with our hosting provider so we had to restart 2 out of 3 nodes(each node is on separate VPS). Current block is 4 000 000. When node 1 and node 2 were restarted they started mining from block 372 instead of the last one 4 000 000.
Why it happened (my guess)
Geth 1.8.22 keeps some data with trie node data in RAM instead of a disk. On graceful node shutdown(for example from console) this trie node data is saved to hard drive from RAM. On forced system shutdown(for example from hosting admin panel) trie node data does not have time to be saved on a hard drive. We had our nodes running for 6 months without any reboot so I think that this trie node data was kept in RAM for the whole time and it was vanished on system reboot(though we still have node 3 which is up and running).
Logs
Here are the logs when I'm trying to run the backup version of one of the nodes:
vladimir#comp:~/Public/projects/ethereum/repro-geth-bug/geth-linux-amd64-1.8.22-7fa3509e$ ./geth --datadir ../opt/ethereum/data/ --networkid 1515 --unlock 0xd6ee38421e1713dd50e888c6d689b82953946bc3 --password ../opt/ethereum/unlock_password --port 30306 --mine
INFO [11-21|17:06:25.374] Maximum peer count ETH=25 LES=0 total=25
INFO [11-21|17:06:25.374] Starting peer-to-peer node instance=Geth/v1.8.22-stable-7fa3509e/linux-amd64/go1.11.5
INFO [11-21|17:06:25.374] Allocated cache and file handles database=/home/vladimir/Public/projects/ethereum/repro-geth-bug/opt/ethereum/data/geth/chaindata cache=512 handles=2048
INFO [11-21|17:06:26.550] Initialised chain configuration config="{ChainID: 1515 Homestead: 1 DAO: <nil> DAOSupport: false EIP150: 2 EIP155: 3 EIP158: 3 Byzantium: 4 Constantinople: 5 ConstantinopleFix: <nil> Engine: clique}"
INFO [11-21|17:06:26.550] Initialising Ethereum protocol versions="[63 62]" network=1515
WARN [11-21|17:06:26.579] Head state missing, repairing chain number=4073749 hash=9bfb53…56d503
INFO [11-21|17:07:45.179] Rewound blockchain to past state number=371 hash=102018…d91947
INFO [11-21|17:07:45.180] Loaded most recent local header number=4073749 hash=9bfb53…56d503 td=8147499 age=2d5h43m
INFO [11-21|17:07:45.180] Loaded most recent local full block number=371 hash=102018…d91947 td=743 age=7mo3w6d
INFO [11-21|17:07:45.180] Loaded most recent local fast block number=4073749 hash=9bfb53…56d503 td=8147499 age=2d5h43m
INFO [11-21|17:07:45.180] Loaded local transaction journal transactions=3 dropped=3
INFO [11-21|17:07:45.180] Regenerated local transaction journal transactions=0 accounts=0
WARN [11-21|17:07:45.180] Blockchain not empty, fast sync disabled
INFO [11-21|17:07:45.623] New local node record seq=6 id=e8c5a9e8848d4e30 ip=127.0.0.1 udp=30306 tcp=30306
INFO [11-21|17:07:45.623] Started P2P networking self=enode://9647000ba2579dd529574b49f472f029839a09257c1bc3ade5135cbbb5f3ceaf1237aff5b6b947d2fa4f218fa24858dc2767bd4b78e082b04c9d013c1482cfa6#127.0.0.1:30306
INFO [11-21|17:07:45.624] IPC endpoint opened url=/home/vladimir/Public/projects/ethereum/repro-geth-bug/opt/ethereum/data/geth.ipc
INFO [11-21|17:07:46.192] Unlocked account address=0xd6ee38421e1713dD50E888c6D689B82953946bC3
INFO [11-21|17:07:46.192] Transaction pool price threshold updated price=1000000000
INFO [11-21|17:07:46.192] Transaction pool price threshold updated price=1000000000
INFO [11-21|17:07:46.192] Etherbase automatically configured address=0xd6ee38421e1713dD50E888c6D689B82953946bC3
INFO [11-21|17:07:46.192] Commit new mining work number=372 sealhash=685e15…2c52df uncles=0 txs=0 gas=0 fees=0 elapsed=75.951µs
INFO [11-21|17:07:46.192] Successfully sealed new block number=372 sealhash=685e15…2c52df hash=0c60ef…f29e6b elapsed=385.27µs
INFO [11-21|17:07:46.192] 🔨 mined potential block number=372 hash=0c60ef…f29e6b
INFO [11-21|17:07:46.193] Commit new mining work number=373 sealhash=337ae5…2b4704 uncles=0 txs=0 gas=0 fees=0 elapsed=222.362µs
INFO [11-21|17:07:47.962] Mapped network port proto=tcp extport=30306 intport=30306 interface="UPNP IGDv1-IP1"
INFO [11-21|17:07:48.391] Mapped network port proto=udp extport=30306 intport=30306 interface="UPNP IGDv1-IP1"
INFO [11-21|17:07:49.625] New local node record seq=7 id=e8c5a9e8848d4e30 ip=128.71.103.50 udp=30306 tcp=30306
INFO [11-21|17:07:51.001] Successfully sealed new block number=373 sealhash=337ae5…2b4704 hash=b67668…81f164 elapsed=4.807s
INFO [11-21|17:07:51.001] 🔨 mined potential block number=373 hash=b67668…81f164
INFO [11-21|17:07:51.002] Commit new mining work number=374 sealhash=c0e9f6…628d51 uncles=0 txs=0 gas=0 fees=0 elapsed=1.434ms
INFO [11-21|17:07:56.001] Successfully sealed new block number=374 sealhash=c0e9f6…628d51 hash=77aae2…9c44e8 elapsed=4.998s
INFO [11-21|17:07:56.001] 🔨 mined potential block number=374 hash=77aae2…9c44e8
INFO [11-21|17:07:56.003] Commit new mining work number=375 sealhash=6f7db7…adca12 uncles=0 txs=0 gas=0 fees=0 elapsed=1.305ms
^CINFO [11-21|17:07:58.483] Got interrupt, shutting down...
INFO [11-21|17:07:58.483] IPC endpoint closed url=/home/vladimir/Public/projects/ethereum/repro-geth-bug/opt/ethereum/data/geth.ipc
INFO [11-21|17:07:58.483] Writing cached state to disk block=374 hash=77aae2…9c44e8 root=e16e04…e93be1
INFO [11-21|17:07:58.483] Persisted trie from memory database nodes=0 size=0.00B time=7.185µs gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B
INFO [11-21|17:07:58.483] Writing cached state to disk block=373 hash=b67668…81f164 root=e16e04…e93be1
INFO [11-21|17:07:58.483] Persisted trie from memory database nodes=0 size=0.00B time=2.571µs gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B
INFO [11-21|17:07:58.484] Writing cached state to disk block=247 hash=7b422a…5f9a62 root=e16e04…e93be1
INFO [11-21|17:07:58.484] Persisted trie from memory database nodes=0 size=0.00B time=2.784µs gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B
INFO [11-21|17:07:58.484] Blockchain manager stopped
INFO [11-21|17:07:58.484] Stopping Ethereum protocol
INFO [11-21|17:07:58.484] Ethereum protocol stopped
INFO [11-21|17:07:58.484] Transaction pool stopped
INFO [11-21|17:07:58.497] Database closed database=/home/vladimir/Public/projects/ethereum/repro-geth-bug/opt/ethereum/data/geth/chaindata
How to fix
The 1st thing that comes to mind is to restart geth nodes(gracefully) via cron everyday so that nodes persist trie node data on the disk.
How to handle UNgraceful system shutdown so that geth node persists data and keeps mining from the latest block on restart?
Please check the full answer: https://github.com/ethereum/go-ethereum/issues/20383#issuecomment-558107815
In short:
geth persists data after 1 hour worth of block processing
if your network is super light (i.e. mostly empty blocks), it takes a very very long time until blocks are flushed from memory to hard drive
currently there is no way to configure the period of persistency rounds in geth
Solution: restart geth periodically so it saves data from RAM to hard drive

How to Configure Mysql 5.5.42 Data folder to Program data file using innosetup

I Have Mysql server 5.5.42 Installed Files, I have successfully Configured & Created Mysql Services, Its Working Good.
My Question is When Im Configuring Mysql Using Innosetup, The Data Folder is Automatically Assigned as in C:\Program Files (x86)\HiMySQL\data but i need to change this Directory to C:\ProgramData\HiMySQL\data
Here is My Code for configuration.
Filename: {pf32}\HiMySQL\bin\MySQLInstanceConfig.exe; Parameters:"-i -q ""-l{pf32}\HiMySQL\mysql_install_log.txt""""-nMySQL Server 5.5"" ""-p{pf32}\HiMySQL\"" -v5.5.23 ""-t{pf32}\HiMySQL\my-template.ini"" ""-c{pf32}\HiMySQL\my.ini"" ServerType=SERVER DatabaseType=MIXED Port=3344 StrictMode=yes ConnectionCount=15 Charset=utf8 ServiceName=HIMySQL AddBinToPath=yes RootPassword=myserver";WorkingDir: {pf32}\HiMySQL\bin\; StatusMsg: Configuring MySQL services; Description: Configuring MySQL Service; Flags: runhidden
Filename: net.exe;Parameters: start HIMySQL;StatusMsg: Starting MySQL Server;Flags: runhidden
Please Help Me Out To Resolve this issue.
my-template.ini file code
# MySQL Server Instance Configuration File
# ----------------------------------------------------------------------
# Generated by the MySQL Server Instance Configuration Wizard
#
#
# Installation Instructions
# ----------------------------------------------------------------------
#
# On Linux you can copy this file to /etc/my.cnf to set global options,
# mysql-data-dir/my.cnf to set server-specific options
# (#localstatedir# for this installation) or to
# ~/.my.cnf to set user-specific options.
#
# On Windows you should keep this file in the installation directory
# of your server (e.g. C:\Program Files\MySQL\MySQL Server X.Y). To
# make sure the server reads the config file use the startup option
# "--defaults-file".
#
# To run run the server from the command line, execute this in a
# command line shell, e.g.
# mysqld --defaults-file="C:\Program Files\MySQL\MySQL Server X.Y\my.ini"
#
# To install the server as a Windows service manually, execute this in a
# command line shell, e.g.
# mysqld --install MySQLXY --defaults-file="C:\Program Files\MySQL\MySQL Server X.Y\my.ini"
#
# And then execute this in a command line shell to start the server, e.g.
# net start MySQLXY
#
#
# Guildlines for editing this file
# ----------------------------------------------------------------------
#
# In this file, you can use all long options that the program supports.
# If you want to know the options a program supports, start the program
# with the "--help" option.
#
# More detailed information about the individual options can also be
# found in the manual.
#
#
# CLIENT SECTION
# ----------------------------------------------------------------------
#
# The following options will be read by MySQL client applications.
# Note that only client applications shipped by MySQL are guaranteed
# to read this section. If you want your own MySQL client program to
# honor these values, you need to specify it as an option during the
# MySQL client library initialization.
#
[client]
port=3339
[mysql]
default-character-set=latin1
# SERVER SECTION
# ----------------------------------------------------------------------
#
# The following options will be read by the MySQL Server. Make sure that
# you have installed the server correctly (see above) so it reads this
# file.
#
[mysqld]
# The TCP/IP Port the MySQL Server will listen on
port=3339
#Path to installation directory. All paths are usually resolved relative to this.
basedir="C:/Program Files (x86)/MYSQLHI"
#Path to the database root
datadir="C:/ProgramData/MYSQLHI/Data/"
# The default character set that will be used when a new schema or table is
# created and no character set is defined
character-set-server=latin1
# The default storage engine that will be used when create new tables when
default-storage-engine=INNODB
# Set the SQL mode to strict
sql-mode="STRICT_TRANS_TABLES,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION"
# The maximum amount of concurrent sessions the MySQL server will
# allow. One of these connections will be reserved for a user with
# SUPER privileges to allow the administrator to login even if the
# connection limit has been reached.
max_connections=100
# Query cache is used to cache SELECT results and later return them
# without actual executing the same query once again. Having the query
# cache enabled may result in significant speed improvements, if your
# have a lot of identical queries and rarely changing tables. See the
# "Qcache_lowmem_prunes" status variable to check if the current value
# is high enough for your load.
# Note: In case your tables change very often or if your queries are
# textually different every time, the query cache may result in a
# slowdown instead of a performance improvement.
query_cache_size=0
# The number of open tables for all threads. Increasing this value
# increases the number of file descriptors that mysqld requires.
# Therefore you have to make sure to set the amount of open files
# allowed to at least 4096 in the variable "open-files-limit" in
# section [mysqld_safe]
table_cache=256
# Maximum size for internal (in-memory) temporary tables. If a table
# grows larger than this value, it is automatically converted to disk
# based table This limitation is for a single table. There can be many
# of them.
tmp_table_size=17M
# How many threads we should keep in a cache for reuse. When a client
# disconnects, the client's threads are put in the cache if there aren't
# more than thread_cache_size threads from before. This greatly reduces
# the amount of thread creations needed if you have a lot of new
# connections. (Normally this doesn't give a notable performance
# improvement if you have a good thread implementation.)
thread_cache_size=8
#*** MyISAM Specific options
# The maximum size of the temporary file MySQL is allowed to use while
# recreating the index (during REPAIR, ALTER TABLE or LOAD DATA INFILE.
# If the file-size would be bigger than this, the index will be created
# through the key cache (which is slower).
myisam_max_sort_file_size=100G
# If the temporary file used for fast index creation would be bigger
# than using the key cache by the amount specified here, then prefer the
# key cache method. This is mainly used to force long character keys in
# large tables to use the slower key cache method to create the index.
myisam_sort_buffer_size=33M
# Size of the Key Buffer, used to cache index blocks for MyISAM tables.
# Do not set it larger than 30% of your available memory, as some memory
# is also required by the OS to cache rows. Even if you're not using
# MyISAM tables, you should still set it to 8-64M as it will also be
# used for internal temporary disk tables.
key_buffer_size=24M
# Size of the buffer used for doing full table scans of MyISAM tables.
# Allocated per thread, if a full scan is needed.
read_buffer_size=64K
read_rnd_buffer_size=256K
# This buffer is allocated when MySQL needs to rebuild the index in
# REPAIR, OPTIMZE, ALTER table statements as well as in LOAD DATA INFILE
# into an empty table. It is allocated per thread so be careful with
# large settings.
sort_buffer_size=256K
#*** INNODB Specific options ***
# Use this option if you have a MySQL server with InnoDB support enabled
# but you do not plan to use it. This will save memory and disk space
# and speed up some things.
#skip-innodb
# Additional memory pool that is used by InnoDB to store metadata
# information. If InnoDB requires more memory for this purpose it will
# start to allocate it from the OS. As this is fast enough on most
# recent operating systems, you normally do not need to change this
# value. SHOW INNODB STATUS will display the current amount used.
innodb_additional_mem_pool_size=2M
# If set to 1, InnoDB will flush (fsync) the transaction logs to the
# disk at each commit, which offers full ACID behavior. If you are
# willing to compromise this safety, and you are running small
# transactions, you may set this to 0 or 2 to reduce disk I/O to the
# logs. Value 0 means that the log is only written to the log file and
# the log file flushed to disk approximately once per second. Value 2
# means the log is written to the log file at each commit, but the log
# file is only flushed to disk approximately once per second.
innodb_flush_log_at_trx_commit=1
# The size of the buffer InnoDB uses for buffering log data. As soon as
# it is full, InnoDB will have to flush it to disk. As it is flushed
# once per second anyway, it does not make sense to have it very large
# (even with long transactions).
innodb_log_buffer_size=1M
# InnoDB, unlike MyISAM, uses a buffer pool to cache both indexes and
# row data. The bigger you set this the less disk I/O is needed to
# access data in tables. On a dedicated database server you may set this
# parameter up to 80% of the machine physical memory size. Do not set it
# too large, though, because competition of the physical memory may
# cause paging in the operating system. Note that on 32bit systems you
# might be limited to 2-3.5G of user level memory per process, so do not
# set it too high.
innodb_buffer_pool_size=45M
# Size of each log file in a log group. You should set the combined size
# of log files to about 25%-100% of your buffer pool size to avoid
# unneeded buffer pool flush activity on log file overwrite. However,
# note that a larger logfile size will increase the time needed for the
# recovery process.
innodb_log_file_size=23M
# Number of threads allowed inside the InnoDB kernel. The optimal value
# depends highly on the application, hardware as well as the OS
# scheduler properties. A too high value may lead to thread thrashing.
innodb_thread_concurrency=8
Innosetup Script code
; Script generated by the Inno Script Studio Wizard.
; SEE THE DOCUMENTATION FOR DETAILS ON CREATING INNO SETUP SCRIPT FILES!
#define MyAppName "HiMySQL"
#define MyAppVersion "1.5"
#define MyAppPublisher "My Company, Inc."
#define MyAppURL "http://www.example.com/"
[Setup]
; NOTE: The value of AppId uniquely identifies this application.
; Do not use the same AppId value in installers for other applications.
; (To generate a new GUID, click Tools | Generate GUID inside the IDE.)
AppId={{C3323AB2-E5D7-4B94-AF8E-C2E71BD468DE}
AppName={#MyAppName}
AppVersion={#MyAppVersion}
;AppVerName={#MyAppName} {#MyAppVersion}
AppPublisher={#MyAppPublisher}
AppPublisherURL={#MyAppURL}
AppSupportURL={#MyAppURL}
AppUpdatesURL={#MyAppURL}
DefaultDirName={pf}\{#MyAppName}
DisableDirPage=yes
DefaultGroupName={#MyAppName}
DisableProgramGroupPage=yes
OutputDir=C:\Users\hyva\Desktop\iss
OutputBaseFilename=setup1
Compression=lzma
SolidCompression=yes
[Languages]
Name: "english"; MessagesFile: "compiler:Default.isl"
[Files]
Source: "C:\Users\hyva\Desktop\m\*"; DestDir: "{pf32}\HiMySQL"; Flags: ignoreversion recursesubdirs createallsubdirs
; NOTE: Don't use "Flags: ignoreversion" on any shared system files
[Run]
Filename: {pf32}\HiMySQL\bin\MySQLInstanceConfig.exe; Parameters:"-i -q ""-l{pf32}\HiMySQL\mysql_install_log.txt"" ""-nMySQL Server 5.5"" ""-p{pf32}\HiMySQL\"" -v5.5.23 ""-t{pf32}\HiMySQL\my-template.ini"" ""-c{pf32}\HiMySQL\my.ini"" ServerType=SERVER DatabaseType=MIXED Port=3339 StrictMode=yes ConnectionCount=15 Charset=utf8 ServiceName=MySQL57 AddBinToPath=yes RootPassword=myserver"; WorkingDir: {pf32}\HiMySQL\bin\; StatusMsg: Configuring MySQL services; Description: Configuring MySQL Service; Flags: runhidden
Filename: net.exe;Parameters: start MySQL57;StatusMsg: Starting MySQL Server;Flags: runhidden
Create a template config file with
datadir: "C:/ProgramData/HiMySQL/data"
And pass a path to that file using -t switch to MySQLInstanceConfig.exe.

Is it possible to check that a particular query opens how many files in MySQL?

I have large number of open files limit in MySQL.
I have set open_files_limit to 150000 but still MySQL uses almost 80% of it.
Also I have low traffic and max concurrent connections around 30 and no query has more than 4 joins.
The files opened by the server are visible in the performance_schema.
See table performance_schema.file_instances.
http://dev.mysql.com/doc/refman/5.5/en/file-instances-table.html
As for tracing which query opens which file, it does not work that way, due to caching in the server itself (table cache, table definition cache).
MySQL shouldn't open that many files, unless you have set a ludicrously large value for the table_cache parameter (the default is 64, the maximum is 512K).
You can reduce the number of open files by issuing the FLUSH TABLES command.
Otherwise, the appropriate value of table_cache can be roughly estimated (in Linux) by running strace -c against all MySQLd threads. You get something like:
# strace -f -c -p $( pidof mysqld )
Process 13598 attached with 22 threads
[ ...pause while it gathers information... ]
^C
Process 13598 detached
...
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
58.82 0.040000 51 780 io_getevents
29.41 0.020000 105 191 93 futex
11.76 0.008000 103 78 select
0.00 0.000000 0 72 stat
0.00 0.000000 0 20 lstat
0.00 0.000000 0 16 lseek
0.00 0.000000 0 16 read
0.00 0.000000 0 9 3 open
0.00 0.000000 0 5 close
0.00 0.000000 0 6 poll
...
------ ----------- ----------- --------- --------- ----------------
...and see whether there's a reasonable difference in impact in open() and close() calls; those are the calls which table_cache affects, and that influence how many open files there are at any given point.
If the impact of open() is negligible, then by all means reduce table_cache. It is mostly needed on slow IOSS'es, and there aren't many of those left around.
If you're running on Windows, you'll have to try and use ProcMon by SysInternals, or some similar tool.
Once you have table_cache to manageable levels, your query that now opens too many files will simply close and re-open many of those same files. You'll perhaps notice an impact on performances, that in all likelihood will be negligible. Chances are that a smaller table cache might actually get you results faster, as fetching an item from a modern, fast IOSS cache may well be faster than searching for it in a really large cache.
If you're into optimizing your server, you may want to look at this article too. The take-away is that as caches go, larger is not always better (it also applies to indexing).
Inspecting a specific query on Linux
On Linux you can use strace (see above) and verify what files are opened and how:
$ sudo strace -f -p $( pidof mysqld ) 2>&1 | grep 'open("'
Meanwhile from a different terminal I run a query, and:
[pid 8894] open("./ecm/db.opt", O_RDONLY) = 39
[pid 8894] open("./ecm/prof2_people.frm", O_RDONLY) = 39
[pid 8894] open("./ecm/prof2_discip.frm", O_RDONLY) = 39
[pid 8894] open("./ecm/prof2_discip.ibd", O_RDONLY) = 19
[pid 8894] open("./ecm/prof2_discip.ibd", O_RDWR) = 19
[pid 8894] open("./ecm/prof2_people.ibd", O_RDONLY) = 20
[pid 8894] open("./ecm/prof2_people.ibd", O_RDWR) = 20
[pid 8894] open("/proc/sys/vm/overcommit_memory", O_RDONLY|O_CLOEXEC) = 39
...these are the files that the query used (*be sure to run the query on a "cold-started" MySQL to prevent caching), and I see that the highest file handle assigned was 39, thus at no point were there more than 40 open files.
The same files can be checked from /proc/$PID/fd or from MySQL:
select * from performance_schema.file_instances where open_count > 1;
but the count from MySQL is slightly shorter, it does not take into account socket descriptors, log files, and temporary files.
This would only be possible by adjusting the source code and add logging on that level.
ALternative: Run a test using this scenario:
You will have to setup an automated test to make this possible:
Log your queries;
Create a script which preloads your heap with a normal dataset (else you are testing against empty memory), take a snapshot of the number of open tables;
Run every query and take snapshot of open tables; (In retrospect) I think you could do this without restarting MySQL every time, so then just every query and record the results. Debugging is tedious work: Not impossible, just really tedious.
Personally I would start different:
Install cacti and percona cacti plugin
Register a week of normal workload
Then hunt down high load queries (slow log > 0.1 second, run through a script to find repeating queries).
Another week monitoring
Then hunt down additional queries with a high repeat count: This is often inefficient code firing a high number of queries where less could be used (like retrieving the keys and then all the values for every key per key (one by one: Happens a lot when programmers use ORM).

sphinx index fail, and ask me to restart the server with query_cache_type=1 to enable it

mysql config my.ini default query_cache_type=0 .
I have already set sql_query_pre = SET SESSION query_cache_type=OFF in sphinx.conf.I think it is not good to turn cache while indexing.But sphinx still asking me to turn on cache...
error detail:
win7 x64, sphinx 2.1.7
I:\sphinx\bin>I:\sphinx\bin\indexer.exe --all --config I:\sphinx\bin\sphinx.conf
Sphinx 2.1.7-id64-release (r4638)
Copyright (c) 2001-2014, Andrew Aksyonoff
Copyright (c) 2008-2014, Sphinx Technologies Inc (http://sphinxsearch.com)
using config file 'I:\sphinx\bin\sphinx.conf'...
indexing index 'test1'...
ERROR: index 'test1': sql_query_pre[1]: Query cache is disabled; restart the server with query_cache_type=1 to enable it
(DSN=mysql://root:***#localhost:3306/test).
total 0 docs, 0 bytes
total 0.018 sec, 0 bytes/sec, 0.00 docs/sec
skipping non-plain index 'rt'...
total 0 reads, 0.000 sec, 0.0 kb/call avg, 0.0 msec/call avg
total 0 writes, 0.000 sec, 0.0 kb/call avg, 0.0 msec/call avg
The 'message' you are receiving is coming from mysql - not from sphinx. indexer just runs the commands as provided and reports/uses the results.
Basically mysql is telling yo the query cache is already disabled. its not enabled globally.
So trying to turn if off for just the (indexing) session, fails, because its not on. If its not enabled in teh first place you cant disable it!
http://www.big.info/2013/04/error-code-1651-query-cache-is-disabled.html
Its telling you NEED to turn it on globally first, before you are ABLE to turn if off.
Maybe mysql could just silently fail to turn it off, rather than giving an error, but thats a different story.
I had a case where I was seeing this error, and it was actually preventing the indexer --all command from generating indices. I went to the XAMPP Control Panel and clicked on the Config button for the MySQL module. This opened the file my.ini in Notepad. I added the following line to the [mysqld] section in the file:
query_cache_type = 1
Then I restarted the MySQL service. The value of query_cache_type was now displayed as ON, and the indexer --all command successfully generated indices.

mytop Perl monitor of MySQL is no more valid in terms of Queries/Questions since MySQLd 5.1.63?

We have Perl mytop of Jeremy Zawodny version 2009-04-06 installed on Debian Squeeze OS, with MySQLd version 5.1.53, using apt-get command. I redo the "apt-get install mytop" that indicates that no newer version of mytop is available.
This version of mytop seems outdated, as it gives systematically very low value of queries done by MySQLd. In fact, it uses the status query to get the total queries since uptime:
SHOW STATUS LIKE 'Questions';
It yields an error result in the new version mysqld. In effect to get mysql total number of queries since uptime, the new mysqld server shifted the number of queries to 'Queries' instead of 'Questions':
SHOW STATUS LIKE 'Queries';
You can see an enormous difference between the two variables by:
mysql> SHOW STATUS LIKE 'Que%';
+---------------+--------+
| Variable_name | Value |
+---------------+--------+
| Queries | 135903 |
| Questions | 160 |
+---------------+--------+
2 rows in set (0.00 sec)
that gives the both values of 'Queries' and 'Questions'.
mytop -uJohnDoe2 -ppassword
Here is the original mytop output:
MySQL on localhost (5.1.63-0+squeeze1-log) up 0+01:42:35 [13:36:44]
Queries: 265.0 qps: 0 Slow: 0.0 Se/In/Up/De(%): 14760/00/00/00
qps now: 0 Slow qps: 0.0 Threads: 5 ( 1/ 5) 1500/00/00/00
Key Efficiency: 100.0% Bps in/out: 0.9/173.8 Now in/out: 8.3/ 1.5k
Id User Host/IP DB Time Cmd Query or State
-- ---- ------- -- ---- --- --------------
28019 root localhost 0 Query show full processlist
....
I copied mytop to mytop.pl, and replaced in the source Perl code the string "Questions" by "Queries", and run
mytop.pl -uJohnDoe2 -ppassword
And the modifed mytop.pl gives a more realistic monitoring:
MySQL on localhost (5.1.63-0+squeeze1-log) up 0+01:42:23 [13:36:32]
Queries: 136.1k qps: 23 Slow: 0.0 Se/In/Up/De(%): 28/00/00/00
qps now: 18 Slow qps: 0.0 Threads: 5 ( 1/ 5) 27/00/00/00
Key Efficiency: 100.0% Bps in/out: 0.1/ 18.3 Now in/out: 8.4/ 1.5k
Id User Host/IP DB Time Cmd Query or State
-- ---- ------- -- ---- --- --------------
30789 root localhost 0 Query show full processlist
....
Have you observed this problem in your system? i.e.
Perl monitor of MySQL is now invalid in terms of Queries/Questions since MySQLd 5.1.63?
ADDED:
After reading the answer of Shlomi Noach, I added this link for modified Perl script file:
mytop.pl.
I have indeed noticed this change, and blogged about it: questions or queries?
Apparently this came as a surprise to other monitoring tools (Innotop, MonYOG) developers.
With regard you case you have two very simple options:
Switch to innotop instead
Modify the source code for mytop and replace Questions with Queries.
The change was made in 5.1.31; in aforementioned post you can also read comment by an Oracle employee.