I have been having a recurring issue with MySQL both on our development machine and our client's production server. What seems to be happening is a memory leak and MySQL is exhausting the server of resources until it eventually crashes. This happens less on the production server (it has 24GB of RAM), but almost nightly on our development server (4GB of RAM).
At the moment, here is the output from top for mysqld:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14572 mysql 15 0 2336m 1.6g 5320 S 0.0 54.9 3:24.21 mysqld
At the moment, memory usage is exactly where I would expect it to be based on how the server is configured. However, it eventually grows to 80%+ memory usage, uses considerable amounts of swap, and, the server eventually will become totally unresponsive.
There is a second issue that we are also seeing, not every day, but at least once a week where InnoDB seems to be getting stuck permanently in sleep mode. There will be hundreds of queries that should take a fraction of a second to execute in show processlist for hours until MySQL is restarted. I do not know if it s a separate problem, or if it is related to this one at all. Yesterday morning, the load on our development server was over 190 with ~120 connections to the database stuck open. This issue seems to be very similar, if not exactly the same, as the problem described here, though it seems like it was fixed in 5.1?
http://venublog.com/2010/07/19/random-pauses-in-mysql-file-handle-serialization/
http://venublog.com/2010/06/07/innodb-in-a-complete-locked-mode/
Both servers run Linux:
[root@development ~]# cat /etc/redhat-release
CentOS release 5.4 (Final)
The production server runs MySQL 5.1.45, the development server is on 5.1.48.
[root@development ~]# mysql -v
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 9855
Server version: 5.1.48-log MySQL Community Server (GPL)
There are three main schemas, the largest of which has 651 tables, the other two are much smaller and used to store misc. data (mostly blobs). All tables are InnoDB using the plugin.
mysql> select @@innodb_version;
+------------------+
| @@innodb_version |
+------------------+
| 1.0.6 |
+------------------+
1 row in set (0.00 sec)
Here are the relevant entries from MySQL's config file, every other setting for buffers / caches is just default:
[mysqld]
thread_stack=819200
thread_cache_size=10
open_files_limit=32768
table-cache=3000
table_definition_cache=3000
max_connections=500
query_cache_type=1
query_cache_size=512M
innodb_buffer_pool_size=18000M
innodb_file_per_table=1
innodb_flush_log_at_trx_commit=2
innodb_flush_method=O_DIRECT
ignore-builtin-innodb
plugin-load=innodb=ha_innodb_plugin.so
plugin_dir=/usr/local/mysql/lib/plugin
Although max_connections is set to 500, at peak on the development server it does not come close to that amount of usage.
Here is the full output of show global status:
+---------------------------------------+------------+
| Variable_name | Value |
+---------------------------------------+------------+
| Aborted_clients | 3 |
| Aborted_connects | 5 |
| Binlog_cache_disk_use | 0 |
| Binlog_cache_use | 75139 |
| Bytes_received | 26998206 |
| Bytes_sent | 1321463317 |
| Com_admin_commands | 1 |
| Com_assign_to_keycache | 0 |
| Com_alter_db | 0 |
| Com_alter_db_upgrade | 0 |
| Com_alter_event | 0 |
| Com_alter_function | 0 |
| Com_alter_procedure | 0 |
| Com_alter_server | 0 |
| Com_alter_table | 0 |
| Com_alter_tablespace | 0 |
| Com_analyze | 0 |
| Com_backup_table | 0 |
| Com_begin | 399 |
| Com_binlog | 0 |
| Com_call_procedure | 13 |
| Com_change_db | 841 |
| Com_change_master | 0 |
| Com_check | 0 |
| Com_checksum | 0 |
| Com_commit | 395 |
| Com_create_db | 0 |
| Com_create_event | 0 |
| Com_create_function | 0 |
| Com_create_index | 0 |
| Com_create_procedure | 0 |
| Com_create_server | 0 |
| Com_create_table | 58 |
| Com_create_trigger | 0 |
| Com_create_udf | 0 |
| Com_create_user | 0 |
| Com_create_view | 0 |
| Com_dealloc_sql | 6 |
| Com_delete | 19 |
| Com_delete_multi | 8 |
| Com_do | 0 |
| Com_drop_db | 0 |
| Com_drop_event | 0 |
| Com_drop_function | 0 |
| Com_drop_index | 0 |
| Com_drop_procedure | 0 |
| Com_drop_server | 0 |
| Com_drop_table | 56 |
| Com_drop_trigger | 0 |
| Com_drop_user | 0 |
| Com_drop_view | 0 |
| Com_empty_query | 0 |
| Com_execute_sql | 6 |
| Com_flush | 0 |
| Com_grant | 0 |
| Com_ha_close | 0 |
| Com_ha_open | 0 |
| Com_ha_read | 0 |
| Com_help | 0 |
| Com_insert | 425 |
| Com_insert_select | 1171 |
| Com_install_plugin | 0 |
| Com_kill | 0 |
| Com_load | 0 |
| Com_load_master_data | 0 |
| Com_load_master_table | 0 |
| Com_lock_tables | 0 |
| Com_optimize | 0 |
| Com_preload_keys | 0 |
| Com_prepare_sql | 6 |
| Com_purge | 0 |
| Com_purge_before_date | 0 |
| Com_release_savepoint | 0 |
| Com_rename_table | 0 |
| Com_rename_user | 0 |
| Com_repair | 0 |
| Com_replace | 75116 |
| Com_replace_select | 0 |
| Com_reset | 0 |
| Com_restore_table | 0 |
| Com_revoke | 0 |
| Com_revoke_all | 0 |
| Com_rollback | 0 |
| Com_rollback_to_savepoint | 0 |
| Com_savepoint | 0 |
| Com_select | 1065706 |
| Com_set_option | 87918 |
| Com_show_authors | 0 |
| Com_show_binlog_events | 0 |
| Com_show_binlogs | 0 |
| Com_show_charsets | 0 |
| Com_show_collations | 0 |
| Com_show_column_types | 0 |
| Com_show_contributors | 0 |
| Com_show_create_db | 15 |
| Com_show_create_event | 0 |
| Com_show_create_func | 0 |
| Com_show_create_proc | 0 |
| Com_show_create_table | 1622 |
| Com_show_create_trigger | 307 |
| Com_show_databases | 3 |
| Com_show_engine_logs | 0 |
| Com_show_engine_mutex | 0 |
| Com_show_engine_status | 0 |
| Com_show_events | 0 |
| Com_show_errors | 0 |
| Com_show_fields | 815 |
| Com_show_function_status | 0 |
| Com_show_grants | 0 |
| Com_show_keys | 981 |
| Com_show_master_status | 393 |
| Com_show_new_master | 0 |
| Com_show_open_tables | 0 |
| Com_show_plugins | 0 |
| Com_show_privileges | 0 |
| Com_show_procedure_status | 0 |
| Com_show_processlist | 0 |
| Com_show_profile | 0 |
| Com_show_profiles | 0 |
| Com_show_slave_hosts | 0 |
| Com_show_slave_status | 0 |
| Com_show_status | 317 |
| Com_show_storage_engines | 0 |
| Com_show_table_status | 811 |
| Com_show_tables | 33 |
| Com_show_triggers | 811 |
| Com_show_variables | 1 |
| Com_show_warnings | 0 |
| Com_slave_start | 0 |
| Com_slave_stop | 0 |
| Com_stmt_close | 8 |
| Com_stmt_execute | 580549 |
| Com_stmt_fetch | 0 |
| Com_stmt_prepare | 9 |
| Com_stmt_reprepare | 0 |
| Com_stmt_reset | 0 |
| Com_stmt_send_long_data | 0 |
| Com_truncate | 0 |
| Com_uninstall_plugin | 0 |
| Com_unlock_tables | 4 |
| Com_update | 105 |
| Com_update_multi | 0 |
| Com_xa_commit | 0 |
| Com_xa_end | 0 |
| Com_xa_prepare | 0 |
| Com_xa_recover | 0 |
| Com_xa_rollback | 0 |
| Com_xa_start | 0 |
| Compression | OFF |
| Connections | 9946 |
| Created_tmp_disk_tables | 1732 |
| Created_tmp_files | 5 |
| Created_tmp_tables | 5414 |
| Delayed_errors | 0 |
| Delayed_insert_threads | 0 |
| Delayed_writes | 0 |
| Flush_commands | 1 |
| Handler_commit | 158361 |
| Handler_delete | 5 |
| Handler_discover | 0 |
| Handler_prepare | 150284 |
| Handler_read_first | 16911 |
| Handler_read_key | 1528472 |
| Handler_read_next | 7834881 |
| Handler_read_prev | 4 |
| Handler_read_rnd | 25936 |
| Handler_read_rnd_next | 3375020 |
| Handler_rollback | 4 |
| Handler_savepoint | 0 |
| Handler_savepoint_rollback | 0 |
| Handler_update | 81949 |
| Handler_write | 190797 |
| Innodb_buffer_pool_pages_data | 95841 |
| Innodb_buffer_pool_pages_dirty | 4 |
| Innodb_buffer_pool_pages_flushed | 20174 |
| Innodb_buffer_pool_pages_free | 1 |
| Innodb_buffer_pool_pages_misc | 158 |
| Innodb_buffer_pool_pages_total | 96000 |
| Innodb_buffer_pool_read_ahead | 17329 |
| Innodb_buffer_pool_read_ahead_evicted | 258 |
| Innodb_buffer_pool_read_requests | 9075239 |
| Innodb_buffer_pool_reads | 134258 |
| Innodb_buffer_pool_wait_free | 0 |
| Innodb_buffer_pool_write_requests | 552753 |
| Innodb_data_fsyncs | 35194 |
| Innodb_data_pending_fsyncs | 0 |
| Innodb_data_pending_reads | 0 |
| Innodb_data_pending_writes | 0 |
| Innodb_data_read | 2485784576 |
| Innodb_data_reads | 135363 |
| Innodb_data_writes | 197224 |
| Innodb_data_written | 779732992 |
| Innodb_dblwr_pages_written | 20174 |
| Innodb_dblwr_writes | 3960 |
| Innodb_have_atomic_builtins | ON |
| Innodb_log_waits | 0 |
| Innodb_log_write_requests | 57062 |
| Innodb_log_writes | 173194 |
| Innodb_os_log_fsyncs | 25079 |
| Innodb_os_log_pending_fsyncs | 0 |
| Innodb_os_log_pending_writes | 0 |
| Innodb_os_log_written | 117595136 |
| Innodb_page_size | 16384 |
| Innodb_pages_created | 20 |
| Innodb_pages_read | 151586 |
| Innodb_pages_written | 20174 |
| Innodb_row_lock_current_waits | 0 |
| Innodb_row_lock_time | 15088 |
| Innodb_row_lock_time_avg | 35 |
| Innodb_row_lock_time_max | 1587 |
| Innodb_row_lock_waits | 423 |
| Innodb_rows_deleted | 5 |
| Innodb_rows_inserted | 15 |
| Innodb_rows_read | 11227369 |
| Innodb_rows_updated | 75123 |
| Key_blocks_not_flushed | 0 |
| Key_blocks_unused | 7241 |
| Key_blocks_used | 103 |
| Key_read_requests | 17309 |
| Key_reads | 134 |
| Key_write_requests | 4631 |
| Key_writes | 0 |
| Last_query_cost | 0.000000 |
| Max_used_connections | 7 |
| Not_flushed_delayed_rows | 0 |
| Open_files | 197 |
| Open_streams | 0 |
| Open_table_definitions | 594 |
| Open_tables | 600 |
| Opened_files | 16819 |
| Opened_table_definitions | 4140 |
| Opened_tables | 4300 |
| Prepared_stmt_count | 0 |
| Qcache_free_blocks | 2 |
| Qcache_free_memory | 536449816 |
| Qcache_hits | 2897 |
| Qcache_inserts | 245 |
| Qcache_lowmem_prunes | 0 |
| Qcache_not_cached | 1054068 |
| Qcache_queries_in_cache | 226 |
| Qcache_total_blocks | 519 |
| Queries | 1445586 |
| Questions | 1238556 |
| Rpl_status | NULL |
| Select_full_join | 102 |
| Select_full_range_join | 0 |
| Select_range | 1125 |
| Select_range_check | 0 |
| Select_scan | 19697 |
| Slave_open_temp_tables | 0 |
| Slave_retried_transactions | 0 |
| Slave_running | OFF |
| Slow_launch_threads | 0 |
| Slow_queries | 4 |
| Sort_merge_passes | 0 |
| Sort_range | 210 |
| Sort_rows | 26144 |
| Sort_scan | 1149 |
| Ssl_accept_renegotiates | 0 |
| Ssl_accepts | 0 |
| Ssl_callback_cache_hits | 0 |
| Ssl_cipher | |
| Ssl_cipher_list | |
| Ssl_client_connects | 0 |
| Ssl_connect_renegotiates | 0 |
| Ssl_ctx_verify_depth | 0 |
| Ssl_ctx_verify_mode | 0 |
| Ssl_default_timeout | 0 |
| Ssl_finished_accepts | 0 |
| Ssl_finished_connects | 0 |
| Ssl_session_cache_hits | 0 |
| Ssl_session_cache_misses | 0 |
| Ssl_session_cache_mode | NONE |
| Ssl_session_cache_overflows | 0 |
| Ssl_session_cache_size | 0 |
| Ssl_session_cache_timeouts | 0 |
| Ssl_sessions_reused | 0 |
| Ssl_used_session_cache_entries | 0 |
| Ssl_verify_depth | 0 |
| Ssl_verify_mode | 0 |
| Ssl_version | |
| Table_locks_immediate | 97728 |
| Table_locks_waited | 0 |
| Tc_log_max_pages_used | 0 |
| Tc_log_page_size | 0 |
| Tc_log_page_waits | 0 |
| Threads_cached | 2 |
| Threads_connected | 5 |
| Threads_created | 7 |
| Threads_running | 2 |
| Uptime | 21254 |
| Uptime_since_flush_status | 21254 |
+---------------------------------------+------------+
We've also adjusted swappiness to attempt to keep it from swapping so much:
[root@development ~]# cat /proc/sys/vm/swappiness
0
We have several crons that run to do a lot of database heavy work at night (backups, etc....), but we have tried running them during the day manually expecting to reproduce the problem, but to no avail.
As far as how we connect to the database from our application, it is fairly standard Perl DBI code using DBD::Mysql. I did find an open bug with a memory leak in the DBD driver if server side prepared statements were enabled on the connection, but that is not applicable to us.
Does anyone have any ideas of what other steps we could take to track down the problem?
At the moment, here is the output from top for mysqld:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14572 mysql 15 0 2336m 1.6g 5320 S 0.0 54.9 3:24.21 mysqld
At the moment, memory usage is exactly where I would expect it to be based on how the server is configured. However, it eventually grows to 80%+ memory usage, uses considerable amounts of swap, and, the server eventually will become totally unresponsive.
There is a second issue that we are also seeing, not every day, but at least once a week where InnoDB seems to be getting stuck permanently in sleep mode. There will be hundreds of queries that should take a fraction of a second to execute in show processlist for hours until MySQL is restarted. I do not know if it s a separate problem, or if it is related to this one at all. Yesterday morning, the load on our development server was over 190 with ~120 connections to the database stuck open. This issue seems to be very similar, if not exactly the same, as the problem described here, though it seems like it was fixed in 5.1?
http://venublog.com/2010/07/19/random-pauses-in-mysql-file-handle-serialization/
http://venublog.com/2010/06/07/innodb-in-a-complete-locked-mode/
Both servers run Linux:
[root@development ~]# cat /etc/redhat-release
CentOS release 5.4 (Final)
The production server runs MySQL 5.1.45, the development server is on 5.1.48.
[root@development ~]# mysql -v
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 9855
Server version: 5.1.48-log MySQL Community Server (GPL)
There are three main schemas, the largest of which has 651 tables, the other two are much smaller and used to store misc. data (mostly blobs). All tables are InnoDB using the plugin.
mysql> select @@innodb_version;
+------------------+
| @@innodb_version |
+------------------+
| 1.0.6 |
+------------------+
1 row in set (0.00 sec)
Here are the relevant entries from MySQL's config file, every other setting for buffers / caches is just default:
[mysqld]
thread_stack=819200
thread_cache_size=10
open_files_limit=32768
table-cache=3000
table_definition_cache=3000
max_connections=500
query_cache_type=1
query_cache_size=512M
innodb_buffer_pool_size=18000M
innodb_file_per_table=1
innodb_flush_log_at_trx_commit=2
innodb_flush_method=O_DIRECT
ignore-builtin-innodb
plugin-load=innodb=ha_innodb_plugin.so
plugin_dir=/usr/local/mysql/lib/plugin
Although max_connections is set to 500, at peak on the development server it does not come close to that amount of usage.
Here is the full output of show global status:
+---------------------------------------+------------+
| Variable_name | Value |
+---------------------------------------+------------+
| Aborted_clients | 3 |
| Aborted_connects | 5 |
| Binlog_cache_disk_use | 0 |
| Binlog_cache_use | 75139 |
| Bytes_received | 26998206 |
| Bytes_sent | 1321463317 |
| Com_admin_commands | 1 |
| Com_assign_to_keycache | 0 |
| Com_alter_db | 0 |
| Com_alter_db_upgrade | 0 |
| Com_alter_event | 0 |
| Com_alter_function | 0 |
| Com_alter_procedure | 0 |
| Com_alter_server | 0 |
| Com_alter_table | 0 |
| Com_alter_tablespace | 0 |
| Com_analyze | 0 |
| Com_backup_table | 0 |
| Com_begin | 399 |
| Com_binlog | 0 |
| Com_call_procedure | 13 |
| Com_change_db | 841 |
| Com_change_master | 0 |
| Com_check | 0 |
| Com_checksum | 0 |
| Com_commit | 395 |
| Com_create_db | 0 |
| Com_create_event | 0 |
| Com_create_function | 0 |
| Com_create_index | 0 |
| Com_create_procedure | 0 |
| Com_create_server | 0 |
| Com_create_table | 58 |
| Com_create_trigger | 0 |
| Com_create_udf | 0 |
| Com_create_user | 0 |
| Com_create_view | 0 |
| Com_dealloc_sql | 6 |
| Com_delete | 19 |
| Com_delete_multi | 8 |
| Com_do | 0 |
| Com_drop_db | 0 |
| Com_drop_event | 0 |
| Com_drop_function | 0 |
| Com_drop_index | 0 |
| Com_drop_procedure | 0 |
| Com_drop_server | 0 |
| Com_drop_table | 56 |
| Com_drop_trigger | 0 |
| Com_drop_user | 0 |
| Com_drop_view | 0 |
| Com_empty_query | 0 |
| Com_execute_sql | 6 |
| Com_flush | 0 |
| Com_grant | 0 |
| Com_ha_close | 0 |
| Com_ha_open | 0 |
| Com_ha_read | 0 |
| Com_help | 0 |
| Com_insert | 425 |
| Com_insert_select | 1171 |
| Com_install_plugin | 0 |
| Com_kill | 0 |
| Com_load | 0 |
| Com_load_master_data | 0 |
| Com_load_master_table | 0 |
| Com_lock_tables | 0 |
| Com_optimize | 0 |
| Com_preload_keys | 0 |
| Com_prepare_sql | 6 |
| Com_purge | 0 |
| Com_purge_before_date | 0 |
| Com_release_savepoint | 0 |
| Com_rename_table | 0 |
| Com_rename_user | 0 |
| Com_repair | 0 |
| Com_replace | 75116 |
| Com_replace_select | 0 |
| Com_reset | 0 |
| Com_restore_table | 0 |
| Com_revoke | 0 |
| Com_revoke_all | 0 |
| Com_rollback | 0 |
| Com_rollback_to_savepoint | 0 |
| Com_savepoint | 0 |
| Com_select | 1065706 |
| Com_set_option | 87918 |
| Com_show_authors | 0 |
| Com_show_binlog_events | 0 |
| Com_show_binlogs | 0 |
| Com_show_charsets | 0 |
| Com_show_collations | 0 |
| Com_show_column_types | 0 |
| Com_show_contributors | 0 |
| Com_show_create_db | 15 |
| Com_show_create_event | 0 |
| Com_show_create_func | 0 |
| Com_show_create_proc | 0 |
| Com_show_create_table | 1622 |
| Com_show_create_trigger | 307 |
| Com_show_databases | 3 |
| Com_show_engine_logs | 0 |
| Com_show_engine_mutex | 0 |
| Com_show_engine_status | 0 |
| Com_show_events | 0 |
| Com_show_errors | 0 |
| Com_show_fields | 815 |
| Com_show_function_status | 0 |
| Com_show_grants | 0 |
| Com_show_keys | 981 |
| Com_show_master_status | 393 |
| Com_show_new_master | 0 |
| Com_show_open_tables | 0 |
| Com_show_plugins | 0 |
| Com_show_privileges | 0 |
| Com_show_procedure_status | 0 |
| Com_show_processlist | 0 |
| Com_show_profile | 0 |
| Com_show_profiles | 0 |
| Com_show_slave_hosts | 0 |
| Com_show_slave_status | 0 |
| Com_show_status | 317 |
| Com_show_storage_engines | 0 |
| Com_show_table_status | 811 |
| Com_show_tables | 33 |
| Com_show_triggers | 811 |
| Com_show_variables | 1 |
| Com_show_warnings | 0 |
| Com_slave_start | 0 |
| Com_slave_stop | 0 |
| Com_stmt_close | 8 |
| Com_stmt_execute | 580549 |
| Com_stmt_fetch | 0 |
| Com_stmt_prepare | 9 |
| Com_stmt_reprepare | 0 |
| Com_stmt_reset | 0 |
| Com_stmt_send_long_data | 0 |
| Com_truncate | 0 |
| Com_uninstall_plugin | 0 |
| Com_unlock_tables | 4 |
| Com_update | 105 |
| Com_update_multi | 0 |
| Com_xa_commit | 0 |
| Com_xa_end | 0 |
| Com_xa_prepare | 0 |
| Com_xa_recover | 0 |
| Com_xa_rollback | 0 |
| Com_xa_start | 0 |
| Compression | OFF |
| Connections | 9946 |
| Created_tmp_disk_tables | 1732 |
| Created_tmp_files | 5 |
| Created_tmp_tables | 5414 |
| Delayed_errors | 0 |
| Delayed_insert_threads | 0 |
| Delayed_writes | 0 |
| Flush_commands | 1 |
| Handler_commit | 158361 |
| Handler_delete | 5 |
| Handler_discover | 0 |
| Handler_prepare | 150284 |
| Handler_read_first | 16911 |
| Handler_read_key | 1528472 |
| Handler_read_next | 7834881 |
| Handler_read_prev | 4 |
| Handler_read_rnd | 25936 |
| Handler_read_rnd_next | 3375020 |
| Handler_rollback | 4 |
| Handler_savepoint | 0 |
| Handler_savepoint_rollback | 0 |
| Handler_update | 81949 |
| Handler_write | 190797 |
| Innodb_buffer_pool_pages_data | 95841 |
| Innodb_buffer_pool_pages_dirty | 4 |
| Innodb_buffer_pool_pages_flushed | 20174 |
| Innodb_buffer_pool_pages_free | 1 |
| Innodb_buffer_pool_pages_misc | 158 |
| Innodb_buffer_pool_pages_total | 96000 |
| Innodb_buffer_pool_read_ahead | 17329 |
| Innodb_buffer_pool_read_ahead_evicted | 258 |
| Innodb_buffer_pool_read_requests | 9075239 |
| Innodb_buffer_pool_reads | 134258 |
| Innodb_buffer_pool_wait_free | 0 |
| Innodb_buffer_pool_write_requests | 552753 |
| Innodb_data_fsyncs | 35194 |
| Innodb_data_pending_fsyncs | 0 |
| Innodb_data_pending_reads | 0 |
| Innodb_data_pending_writes | 0 |
| Innodb_data_read | 2485784576 |
| Innodb_data_reads | 135363 |
| Innodb_data_writes | 197224 |
| Innodb_data_written | 779732992 |
| Innodb_dblwr_pages_written | 20174 |
| Innodb_dblwr_writes | 3960 |
| Innodb_have_atomic_builtins | ON |
| Innodb_log_waits | 0 |
| Innodb_log_write_requests | 57062 |
| Innodb_log_writes | 173194 |
| Innodb_os_log_fsyncs | 25079 |
| Innodb_os_log_pending_fsyncs | 0 |
| Innodb_os_log_pending_writes | 0 |
| Innodb_os_log_written | 117595136 |
| Innodb_page_size | 16384 |
| Innodb_pages_created | 20 |
| Innodb_pages_read | 151586 |
| Innodb_pages_written | 20174 |
| Innodb_row_lock_current_waits | 0 |
| Innodb_row_lock_time | 15088 |
| Innodb_row_lock_time_avg | 35 |
| Innodb_row_lock_time_max | 1587 |
| Innodb_row_lock_waits | 423 |
| Innodb_rows_deleted | 5 |
| Innodb_rows_inserted | 15 |
| Innodb_rows_read | 11227369 |
| Innodb_rows_updated | 75123 |
| Key_blocks_not_flushed | 0 |
| Key_blocks_unused | 7241 |
| Key_blocks_used | 103 |
| Key_read_requests | 17309 |
| Key_reads | 134 |
| Key_write_requests | 4631 |
| Key_writes | 0 |
| Last_query_cost | 0.000000 |
| Max_used_connections | 7 |
| Not_flushed_delayed_rows | 0 |
| Open_files | 197 |
| Open_streams | 0 |
| Open_table_definitions | 594 |
| Open_tables | 600 |
| Opened_files | 16819 |
| Opened_table_definitions | 4140 |
| Opened_tables | 4300 |
| Prepared_stmt_count | 0 |
| Qcache_free_blocks | 2 |
| Qcache_free_memory | 536449816 |
| Qcache_hits | 2897 |
| Qcache_inserts | 245 |
| Qcache_lowmem_prunes | 0 |
| Qcache_not_cached | 1054068 |
| Qcache_queries_in_cache | 226 |
| Qcache_total_blocks | 519 |
| Queries | 1445586 |
| Questions | 1238556 |
| Rpl_status | NULL |
| Select_full_join | 102 |
| Select_full_range_join | 0 |
| Select_range | 1125 |
| Select_range_check | 0 |
| Select_scan | 19697 |
| Slave_open_temp_tables | 0 |
| Slave_retried_transactions | 0 |
| Slave_running | OFF |
| Slow_launch_threads | 0 |
| Slow_queries | 4 |
| Sort_merge_passes | 0 |
| Sort_range | 210 |
| Sort_rows | 26144 |
| Sort_scan | 1149 |
| Ssl_accept_renegotiates | 0 |
| Ssl_accepts | 0 |
| Ssl_callback_cache_hits | 0 |
| Ssl_cipher | |
| Ssl_cipher_list | |
| Ssl_client_connects | 0 |
| Ssl_connect_renegotiates | 0 |
| Ssl_ctx_verify_depth | 0 |
| Ssl_ctx_verify_mode | 0 |
| Ssl_default_timeout | 0 |
| Ssl_finished_accepts | 0 |
| Ssl_finished_connects | 0 |
| Ssl_session_cache_hits | 0 |
| Ssl_session_cache_misses | 0 |
| Ssl_session_cache_mode | NONE |
| Ssl_session_cache_overflows | 0 |
| Ssl_session_cache_size | 0 |
| Ssl_session_cache_timeouts | 0 |
| Ssl_sessions_reused | 0 |
| Ssl_used_session_cache_entries | 0 |
| Ssl_verify_depth | 0 |
| Ssl_verify_mode | 0 |
| Ssl_version | |
| Table_locks_immediate | 97728 |
| Table_locks_waited | 0 |
| Tc_log_max_pages_used | 0 |
| Tc_log_page_size | 0 |
| Tc_log_page_waits | 0 |
| Threads_cached | 2 |
| Threads_connected | 5 |
| Threads_created | 7 |
| Threads_running | 2 |
| Uptime | 21254 |
| Uptime_since_flush_status | 21254 |
+---------------------------------------+------------+
We've also adjusted swappiness to attempt to keep it from swapping so much:
[root@development ~]# cat /proc/sys/vm/swappiness
0
We have several crons that run to do a lot of database heavy work at night (backups, etc....), but we have tried running them during the day manually expecting to reproduce the problem, but to no avail.
As far as how we connect to the database from our application, it is fairly standard Perl DBI code using DBD::Mysql. I did find an open bug with a memory leak in the DBD driver if server side prepared statements were enabled on the connection, but that is not applicable to us.
Does anyone have any ideas of what other steps we could take to track down the problem?