To create new DB connection or not? - mysql
I'm running a cron job (every 15 minutes) which takes about a minute to execute. It makes lots of API calls and stores data to the database.
Right now I create a mysql connection at the beginning and use the same connection through out the code. Most of the time is spent making the API calls.
Will it be more efficient to create a new database connection only when it's time to store the data (below)?
Kill the last connection
Wait for API call to complete
Create new DB connection
Execute query
Goto 1
[Edit] Here's the MYSQL report. I'm new to mysql - is there any reason to re-connect to DB based on the following report?
1 MySQL 5.1.26-rc-5.1.26r uptime 0 1:8:58 Tue Jun 15 21:25:03 2010
2
3 __ Key _________________________________________________________________
4 Buffer used 33.00k of 24.00M %Used: 0.13
5 Current 4.52M %Usage: 18.84
6 Write hit 33.33%
7 Read hit 69.16%
8
9 __ Questions ___________________________________________________________
10 Total 1.75k 0.4/s
11 COM_QUIT 319.92k 77.3/s %Total: 18312.
12 -Unknown 319.90k 77.3/s 18311.
13 DMS 1.53k 0.4/s 87.58
14 Com_ 199 0.0/s 11.39
15 QC Hits 1 0.0/s 0.06
16 Slow 144 0.0/s 8.24 %DMS: 9.41
17 DMS 1.53k 0.4/s 87.58
18 SELECT 1.22k 0.3/s 69.83 79.74
19 INSERT 155 0.0/s 8.87 10.13
20 UPDATE 155 0.0/s 8.87 10.13
21 REPLACE 0 0/s 0.00 0.00
22 DELETE 0 0/s 0.00 0.00
23 Com_ 199 0.0/s 11.39
24 check 86 0.0/s 4.92
25 show_status 41 0.0/s 2.35
26 set_option 23 0.0/s 1.32
27
28 __ SELECT and Sort _____________________________________________________
29 Scan 653 0.2/s %SELECT: 53.52
30 Range 0 0/s 0.00
31 Full join 0 0/s 0.00
32 Range check 0 0/s 0.00
33 Full rng join 0 0/s 0.00
34 Sort scan 0 0/s
35 Sort range 590 0.1/s
36 Sort mrg pass 0 0/s
37
38 __ Query Cache _________________________________________________________
39 Memory usage 43.57k of 12.00M %Used: 0.35
40 Block Fragmnt 25.35%
41 Hits 1 0.0/s
42 Inserts 916 0.2/s
43 Insrt:Prune 916:1 0.2/s
44 Hit:Insert 0.00:1
45
46 __ Table Locks _________________________________________________________
47 Waited 0 0/s %Total: 0.00
48 Immediate 1.65k 0.4/s
49
50 __ Tables ______________________________________________________________
51 Open 47 of 1024 %Cache: 4.59
52 Opened 54 0.0/s
53
54 __ Connections _________________________________________________________
55 Max used 3 of 60 %Max: 5.00
56 Total 319.92k 77.3/s
57
58 __ Created Temp ________________________________________________________
59 Disk table 2 0.0/s
60 Table 28 0.0/s
61 File 5 0.0/s
62
63 __ Threads _____________________________________________________________
64 Running 3 of 3
65 Cached 0 of 4 %Hit: 100
66 Created 3 0.0/s
67 Slow 0 0/s
68
69 __ Aborted _____________________________________________________________
70 Clients 0 0/s
71 Connects 319.86k 77.3/s
72
73 __ Bytes _______________________________________________________________
74 Sent 52.36M 12.7k/s
75 Received 23.17M 5.6k/s
It's rarely advantageous to drop connections and re-establish them. Making a connection to a DB is normally a fairly heavy process. Lots of apps create connection pools just to avoid this: make some reasonable number of connections and then keep them for long periods of time, maybe even forever, letting each thread or user take connections from the pool when you need them and then giving them back.
If you're having a problem with connections getting orphaned -- the query fails and you never manage to free the connection -- the real solution is to implement proper exception handling so that doesn't happen.
If you have one thread sitting on an inactive connection while other threads are failing because the database has hit its connection maximum, then, yes, you need to look at freeing up connections.
Side note: MySQL claims that it can make connections very quickly, so that this is less of an issue than for other database engines. I have never benchmarked this.
It's a bit hard to give an opinion considering that we have no idea what happens in (2).
Remember that the first rule of optimization is: "Don't do it". In the sense that unless you have good reasons (DB slow for other users, CPU maxed during your cron process and so on) to address the performance problem maybe it's better not to do anything.
If you do have some reason to improve the efficiency of the program, then you will have some hard numbers to compare against (example: your cron batch takes so long that you had to skip some runs, or it ends too late to satisfy user requirements, or fills up the rollback structures etc.) and you can simply apply your modification in your test environment (it looks like a simple fix, unless you forgot to tell us that it would be very complicated to implement) and see if it improves what you have measured and found lacking at the start.
I am sorry but "I wonder if this could be more efficient" without having an idea of what problem you are really trying to address is a recipe for problems.
If the bottle neck is that you do not have enough free slots to connect to the DB then, yes, close the connection when possible.
Otherwise, use it and reuse it (at least in the same request).
Related
MySQL Percentage Rank without the function
I am forced to create a percent rank on a number of columns in a database table and I am really struggling with this. Usually I would use the PERCENT_RANK function but I am forced to use conventional query as our MySQL version doesn't offer that function. I have a table that contains the columns: UID Total-Orders-Placed Last-Order-Date-Diff 12884 8 351 10985 11 106 30613 3 43 30820 2 134 23421 9 76 I would like to add 2 ranking columns as below: UID Total-Orders-Placed Last-Order-Date-Diff rec_rank freq_rank 12884 8 351 0.34 0.86 10985 11 106 0.64 0.91 30613 3 43 0.85 0.59 30820 2 134 0.57 0.40 23421 9 76 0.77 0.88 In reality I have thousands of rows and additional columns but that's the gist. I have been able to do it perfectly in Excel but I am really struggling to convert into Query/Views in our MySQL Database so our data can be viewed realtime. I have tried PERCENT_RANK() but as I mention above this function isn't available to us. I have tried the queries discussed here without too much success yet: http://code.openark.org/blog/mysql/sql-ranking-without-self-join Any help writing the code or to give me a better understanding of it would really help.
Delete and Insert takes long time in MySQL(InnoDB)
I've set of numbers and to each number in it there are few numbers associated with it. So I store it in a table like this: NUMBERS ASSOCIATEDNUMBERS 1 3 1 7 1 8 2 11 2 7 7 9 8 13 11 17 14 18 17 11 17 18 Thus it's a number which has many associated numbers and vice versa. Both columns are indexed. (Thus enabling me to find number and its associated numbers and vice versa) My create table looks like this: CREATE TABLE `TABLE_B` ( `NUMBERS` bigint(20) unsigned NOT NULL, `ASSOCIATEDNUMBERS` bigint(20) unsigned NOT NULL, UNIQUE KEY `unique_number_associatednumber_constraint` (`NUMBERS`,`ASSOCIATEDNUMBERS`), KEY `fk_AssociatedNumberConstraint` (`ASSOCIATEDNUMBERS`), CONSTRAINT `fk_AssociatedNumberConstraint` FOREIGN KEY (`ASSOCIATEDNUMBERS`) REFERENCES `table_a` (`SRNO`), CONSTRAINT `fk_NumberConstraint` FOREIGN KEY (`NUMBERS`) REFERENCES `table_a`` (`SRNO`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 Here TABLE_A has column SRNO which is AUTO_INCREMENT PRIMARY KEY and is first column in the table. (As per MySQL manual I haven't defined indexes on TABLE_B.NUMBERS and TABLE_B.ASSOCIATEDNUMBERS as foreign key constraints defines it automatically) PROBLEM: Whenever I need to change ASSOCIATEDNUMBERS for a number (in `NUMBERS') I just delete existing rows for that number from the table: DELETE FROM TABLE_B WHERE NUMBERS= ? and then insert rows for new set of ASSOCIATEDNUMBERS: INSERT INTO TABLE_B (NUMBERS, ASSOCIATEDNUMBERS) VALUES ( ?, ?), (?, ?), (?, ?), ... However, this takes long time. Especially when in my multi-threaded application I open multiple connections (each per thread) to the database, each running above two queries (but each with different number). For example, if I open 40 connections, each connection to delete existing and insert 250 new associated numbers, it takes upto 10 to 15 seconds. If I increase number of connections, the time also increases. Other Information: SHOW GLOBAL STATUS LIKE 'Threads_running'; Shows upto 40 threads. Innodb parameters: innodb_adaptive_flushing, ON innodb_adaptive_flushing_lwm, 10 innodb_adaptive_hash_index, ON innodb_adaptive_max_sleep_delay, 150000 innodb_additional_mem_pool_size, 2097152 innodb_api_bk_commit_interval, 5 innodb_api_disable_rowlock, OFF innodb_api_enable_binlog, OFF innodb_api_enable_mdl, OFF innodb_api_trx_level, 0 innodb_autoextend_increment, 64 innodb_autoinc_lock_mode, 1 innodb_buffer_pool_dump_at_shutdown, OFF innodb_buffer_pool_dump_now, OFF innodb_buffer_pool_filename, ib_buffer_pool innodb_buffer_pool_instances, 8 innodb_buffer_pool_load_abort, OFF innodb_buffer_pool_load_at_startup, OFF innodb_buffer_pool_load_now, OFF innodb_buffer_pool_size, 1073741824 innodb_change_buffer_max_size, 25 innodb_change_buffering, all innodb_checksum_algorithm, crc32 innodb_checksums, ON innodb_cmp_per_index_enabled, OFF innodb_commit_concurrency, 0 innodb_compression_failure_threshold_pct, 5 innodb_compression_level, 6 innodb_compression_pad_pct_max, 50 innodb_concurrency_tickets, 5000 innodb_data_file_path, ibdata1:12M:autoextend innodb_data_home_dir, innodb_disable_sort_file_cache, OFF innodb_doublewrite, ON innodb_fast_shutdown, 1 innodb_file_format, Antelope innodb_file_format_check, ON innodb_file_format_max, Antelope innodb_file_per_table, ON innodb_flush_log_at_timeout, 1 innodb_flush_log_at_trx_commit, 2 innodb_flush_method, normal innodb_flush_neighbors, 1 innodb_flushing_avg_loops, 30 innodb_force_load_corrupted, OFF innodb_force_recovery, 0 innodb_ft_aux_table, innodb_ft_cache_size, 8000000 innodb_ft_enable_diag_print, OFF innodb_ft_enable_stopword, ON innodb_ft_max_token_size, 84 innodb_ft_min_token_size, 3 innodb_ft_num_word_optimize, 2000 innodb_ft_result_cache_limit, 2000000000 innodb_ft_server_stopword_table, innodb_ft_sort_pll_degree, 2 innodb_ft_total_cache_size, 640000000 innodb_ft_user_stopword_table, innodb_io_capacity, 200 innodb_io_capacity_max, 2000 innodb_large_prefix, OFF innodb_lock_wait_timeout, 50 innodb_locks_unsafe_for_binlog, OFF innodb_log_buffer_size, 268435456 innodb_log_compressed_pages, ON innodb_log_file_size, 262144000 innodb_log_files_in_group, 2 innodb_log_group_home_dir, .\ innodb_lru_scan_depth, 1024 innodb_max_dirty_pages_pct, 75 innodb_max_dirty_pages_pct_lwm, 0 innodb_max_purge_lag, 0 innodb_max_purge_lag_delay, 0 innodb_mirrored_log_groups, 1 innodb_monitor_disable, innodb_monitor_enable, innodb_monitor_reset, innodb_monitor_reset_all, innodb_old_blocks_pct, 37 innodb_old_blocks_time, 1000 innodb_online_alter_log_max_size, 134217728 innodb_open_files, 300 innodb_optimize_fulltext_only, OFF innodb_page_size, 16384 innodb_print_all_deadlocks, OFF innodb_purge_batch_size, 300 innodb_purge_threads, 1 innodb_random_read_ahead, OFF innodb_read_ahead_threshold, 56 innodb_read_io_threads, 64 innodb_read_only, OFF innodb_replication_delay, 0 innodb_rollback_on_timeout, OFF innodb_rollback_segments, 128 innodb_sort_buffer_size, 1048576 innodb_spin_wait_delay, 6 innodb_stats_auto_recalc, ON innodb_stats_method, nulls_equal innodb_stats_on_metadata, OFF innodb_stats_persistent, ON innodb_stats_persistent_sample_pages, 20 innodb_stats_sample_pages, 8 innodb_stats_transient_sample_pages, 8 innodb_status_output, OFF innodb_status_output_locks, OFF innodb_strict_mode, OFF innodb_support_xa, ON innodb_sync_array_size, 1 innodb_sync_spin_loops, 30 innodb_table_locks, ON innodb_thread_concurrency, 0 innodb_thread_sleep_delay, 10000 innodb_undo_directory, . innodb_undo_logs, 128 innodb_undo_tablespaces, 0 innodb_use_native_aio, OFF innodb_use_sys_malloc, ON innodb_version, 5.6.28 innodb_write_io_threads, 16 UPDATE: Here is "SHOW ENGINE InnoDB STATUS" output: http://pastebin.com/raw/E3rK4Pu5 UPDATE2: The reason behind this was somewhere else and not actually DB. Some other function in my code was eating lots of CPU causing MySQL (which runs on same machine) to go slow. Thanks for all your answers and help.
It seems you are acquiring lock on the row/table before deleting/inserting so that's why it is causing you the issue. Check SELECT * from information_schema.GLOBAL_VARIABLES; Also, check locks on table using SHOW OPEN TABLES from <database name> where In_use>0 and lock type using SELECT * FROM INFORMATION_SCHEMA.INNODB_LOCKS So when you run the query add these on watch and you can also use tee command to store it in file. So, there is one more thing that can cause this, although you have indexed the column, but there are limitations to mysql if data Read this for limitation. Read this to install watch http://osxdaily.com/2010/08/22/install-watch-command-on-os-x/ and run watch script with time delay of 1 second and then run mysql queries in watch, if you want to store it in file use tee command and then write output on file. You can tail to get the data in file
SSIS Flat File Export - Only need certain rows from file
I am using SQL Server 2012 and am trying to import a flat file and store it into the database. The problem I am having is that I only need certain rows in a file that contains much more data. Here is an example from part of a file which I am trying to import. 12/02/2015 09:47:44:917 Rx: Message Header: Ver: 1, MsgType: 1, MsgId: 3 Status: 0x00 TranId: 6, Data ByteCount: 55 Data: 86 A6 4E 0B 6A 64 54 2E 00 50 00 02 00 00 60 1A E0 AD 10 12 BF 07 56 54 20 31 32 42 46 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 CB Rx Cmd=84, Rsp code=0x00, Device Status=0x50 Sub-Device Index=2 IO Card=0 Channel=0 Manufacturer ID=24602 The only values I care about are ones which are delimited by = (Sub-Device Index, IO Card, etc.). In this example, how can I import the file in a way that the value 2 gets inserted into a column for Sub-Device Index, 0 inserted into a column for IO Card, 0 for Channel, etc.?
You can do this with a script task that reads and parses the file, and either populates SSIS variables for use with an Execute SQL task, or the script task can insert them directly into the database.
Mysql search for exact number in field
in my MySql DB, i have column containing that kind of value 63 61 57 52 50 47 46 44 43 34 33 27 23 22 21 10 5 3 2 1 Those numbers are separated by tab. Impossible to get the good result for a simple query that would aid something like this SELECT * FROM mytable WHERE mycolumn = 63 I'm not sure if "=" is the good method, i've also tried LIKE, IN and even FIND_IN_SET I need some help :)
Want to Calculate Hightest Pay Value For a table?
I want a list of employees who have worked on the activity that has the highest Total Pay value. don't use code such as …where actid = 151…ect • Note: Total Pay worked for an activity is the sum of the (Total Hours Worked * matching Hourly Rate) (e.g. Total Pay for Activity 151 is 10.5 hrs # $50.75 + 11.5 hrs # $25 + 3hrs # $33,) You must use a subquery in your solution. ACTID HRSWORKED HOURLYRATE Total Pay 163 10 45.5 455 163 8 45.5 364 163 6 45.5 273 151 5 50.75 253.75 151 5.5 50.75 279.125 155 10 30 300 155 10 30 300 165 20 25 500 155 10 30 300 155 8 27 216 151 11.5 25 287.5 151 1 33 33 151 1 33 33 151 1 33 33 You time and effort much appreciated. Thanks !!
Without knowledge of the schema, I can only provide a possible sketch (you'll have to compute total pay and provide all necessary JOINs and predicates): SELECT DISTINCT(employee id) -- reconfigure if more then just employee id FROM <table(s)> [WHERE...] { WHERE | AND } total pay = (SELECT MAX(total pay) FROM <table(s)> [WHERE...]); I used DISTINCT because it's possible to have more than one activity with the same MAX value and overlapping employees. If you're including ACTID in the output, then you won't need DISTINCT because the same employee shouldn't be on a project twice (unless they are tracked by roles on a project in which case a single employee might have multiple roles - it all depends on the data set).