#+TITLE:     EP Stats
#+AUTHOR:    Dustin Sallings
#+EMAIL:     dustin@spy.net
#+DATE:      2010-02-08 Mon
#+DESCRIPTION:
#+KEYWORDS:
#+LANGUAGE:  en
#+OPTIONS:   H:3 num:t toc:t \n:nil @:t ::t |:t ^:nil -:t f:t *:t <:t
#+OPTIONS:   TeX:t LaTeX:nil skip:nil d:nil todo:t pri:nil tags:not-in-toc
#+INFOJS_OPT: view:nil toc:nil ltoc:t mouse:underline buttons:0 path:http://orgmode.org/org-info.js
#+EXPORT_SELECT_TAGS: export
#+EXPORT_EXCLUDE_TAGS: noexport
#+LINK_UP:
#+LINK_HOME:
#+STYLE:  <link rel="stylesheet" type="text/css" href="myorg.css" />

* Getting Started

For introductory information on stats within membase, start with the
[[http://wiki.membase.org/display/membase/Membase%2BStatistics][membase wiki stats page]].

* Stats Definitions

** Toplevel Stats

| Stat                          | Description                                |
|-------------------------------+--------------------------------------------|
| ep_version                    | Version number of ep_engine.               |
| ep_storage_age                | Seconds since most recently                |
|                               | stored object was initially queued.        |
| ep_storage_age_highwat        | ep_storage_age high water mark             |
| ep_min_data_age               | Minimum data age setting.                  |
| ep_queue_age_cap              | Queue age cap setting.                     |
| ep_max_txn_size               | Max number of updates per transaction.     |
| ep_data_age                   | Seconds since most recently                |
|                               | stored object was modified.                |
| ep_data_age_highwat           | ep_data_age high water mark                |
| ep_too_young                  | Number of times an object was              |
|                               | not stored due to being too young.         |
| ep_too_old                    | Number of times an object was              |
|                               | stored after being dirty too long.         |
| ep_total_enqueued             | Total number of items queued for           |
|                               | persistence.                               |
| ep_total_new_items            | Total number of persisted new items.       |
| ep_total_del_items            | Total number of persisted deletions.       |
| ep_total_persisted            | Total number of items persisted.           |
| ep_item_flush_failed          | Number of times an item failed to flush    |
|                               | due to storage errors.                     |
| ep_item_commit_failed         | Number of times a transaction failed to    |
|                               | commit due to storage errors.              |
| ep_item_begin_failed          | Number of times a transaction failed to    |
|                               | start due to storage errors.               |
| ep_expired                    | Number of times an item was expired.       |
| ep_item_flush_expired         | Number of times an item is not flushed     |
|                               | due to the expiry of the item              |
| ep_queue_size                 | Number of items queued for storage.        |
| ep_flusher_todo               | Number of items remaining to be written.   |
| ep_flusher_state              | Current state of the flusher thread.       |
| ep_commit_num                 | Total number of write commits.             |
| ep_commit_time                | Number of seconds of most recent commit.   |
| ep_commit_time_total          | Cumulative seconds spent committing.       |
| ep_vbucket_del                | Number of vbucket deletion events.         |
| ep_vbucket_del_fail           | Number of failed vbucket deletion events.  |
| ep_vbucket_del_max_walltime   | Max wall time (µs) spent by deleting       |
|                               | a vbucket                                  |
| ep_vbucket_del_total_walltime | Total wall time (µs) spent by deleting     |
|                               | vbuckets                                   |
| ep_vbucket_del_avg_walltime   | Avg wall time (µs) spent by deleting       |
|                               | a vbucket                                  |
| ep_flush_preempts             | Num of flush early exits for read reqs.    |
| ep_flush_duration             | Number of seconds of most recent flush.    |
| ep_flush_duration_total       | Cumulative seconds spent flushing.         |
| ep_flush_duration_highwat     | ep_flush_duration high water mark.         |
| ep_flush_all                  | True if disk flush_all is scheduled        |
| curr_items                    | Num items in active vbuckets.              |
| curr_items_tot                | Num current items including those not      |
|                               | active (replica, dead and pending states)  |
| ep_kv_size                    | Memory used to store keys and values.      |
| ep_overhead                   | Extra memory used by rep queues, etc..     |
| ep_max_data_size              | Max amount of data allowed in memory.      |
| ep_mem_low_wat                | Low water mark for auto-evictions.         |
| ep_mem_high_wat               | High water mark for auto-evictions.        |
| ep_total_cache_size           | The total size of all items in the cache   |
| ep_oom_errors                 | Number of times unrecoverable OOMs         |
|                               | happened while processing operations       |
| ep_tmp_oom_errors             | Number of times temporary OOMs             |
|                               | happened while processing operations       |
| ep_bg_fetched                 | Number of items fetched from disk.         |
| ep_tap_bg_fetched             | Number of tap disk fetches                 |
| ep_tap_bg_fetch_requeued      | Number of times a tap bg fetch task is     |
|                               | requeued.                                  |
| ep_num_pager_runs             | Number of times we ran pager loops         |
|                               | to seek additional memory.                 |
| ep_num_expiry_pager_runs      | Number of times we ran expiry pager loops  |
|                               | to purge expired items from memory/disk    |
| ep_num_checkpoint_remover_runs| Number of times we ran checkpoint remover  |
|                               | to remove closed unreferenced checkpoints. |
| ep_items_rm_from_checkpoints  | Number of items removed from closed        |
|                               | unreferenced checkpoints.                  |
| ep_num_value_ejects           | Number of times item values got ejected    |
|                               | from memory to disk                        |
| ep_num_eject_replicas         | Number of times replica item values got    |
|                               | ejected from memory to disk                |
| ep_num_eject_failures         | Number of items that could not be ejected  |
| ep_num_not_my_vbuckets        | Number of times Not My VBucket exception   |
|                               | happened during runtime                    |
| ep_warmup_thread              | Warmup thread status.                      |
| ep_warmed_up                  | Number of items warmed up.                 |
| ep_warmup_dups                | Duplicates encountered during warmup.      |
| ep_warmup_oom                 | OOMs encountered during warmup.            |
| ep_warmup_time                | Time (µs) spent by warming data.           |
| ep_tap_keepalive              | Tap keepalive time.                        |
| ep_dbname                     | DB path.                                   |
| ep_dbinit                     | Number of seconds to initialize DB.        |
| ep_dbshards                   | Number of shards for db store              |
| ep_db_strategy                | SQLite db strategy                         |
| ep_warmup                     | true if warmup is enabled.                 |
| ep_io_num_read                | Number of io read operations               |
| ep_io_num_write               | Number of io write operations              |
| ep_io_read_bytes              | Number of bytes read (key + values)        |
| ep_io_write_bytes             | Number of bytes written (key + values)     |
| ep_pending_ops                | Number of ops awaiting pending vbuckets    |
| ep_pending_ops_total          | Total blocked pending ops since reset      |
| ep_pending_ops_max            | Max ops seen awaiting 1 pending vbucket    |
| ep_pending_ops_max_duration   | Max time (µs) used waiting on pending      |
|                               | vbuckets                                   |
| ep_bg_num_samples             | The number of samples included in the avg  |
| ep_bg_min_wait                | The shortest time (µs) in the wait queue   |
| ep_bg_max_wait                | The longest time (µs) in the wait queue    |
| ep_bg_wait_avg                | The average wait time (µs) for an item     |
|                               | before it is serviced by the dispatcher    |
| ep_bg_min_load                | The shortest load time (µs)                |
| ep_bg_max_load                | The longest load time (µs)                 |
| ep_bg_load_avg                | The average time (µs) for an item to be    |
|                               | loaded from the persistence layer          |
| ep_num_non_resident           | The number of non-resident items           |
| ep_num_active_non_resident    | Number of non-resident items in active     |
|                               | vbuckets.                                  |
| ep_store_max_concurrency      | Maximum allowed concurrency at the storage |
|                               | layer.                                     |
| ep_store_max_readers          | Maximum number of concurrent read-only.    |
|                               | storage threads.                           |
| ep_store_max_readwrite        | Maximum number of concurrent read/write    |
|                               | storage threads.                           |
| ep_db_cleaner_status          | Status of database cleaner that cleans up  |
|                               | invalid items with old vbucket versions    |
| ep_bg_wait                    | The total elapse time for the wait queue   |
| ep_bg_load                    | The total elapse time for items to be      |
|                               | loaded from the persistence layer          |
| ep_latency_get_cmd            | The total elapse time for get command      |
| ep_latency_store_cmd          | The total elapse time for store command    |
| ep_latency_arith_cmd          | The total eplase time for arith command    |
| ep_onlineupdate               | True if engine is in online updated mode   |
| ep_onlineupdate_revert_add    | Number of reverted newly added items       |
| ep_onlineupdate_revert_delete | Number of reverted deleted items           |
| ep_onlineupdate_revert_update | Number of reverted updated items           |

** vBucket total stats

| Stat                          | Description                                |
|-------------------------------+--------------------------------------------|
| ep_vb_total                   | Total vBuckets (count)                     |
| curr_items_tot                | Total number of items                      |
| curr_items                    | Number of active items in memory           |
| vb_dead_num                   | Number of dead vBuckets                    |
| ep_diskqueue_items            | Total items in disk queue                  |
| ep_diskqueue_memory           | Total memory used in disk queue            |
| ep_diskqueue_fill             | Total enqueued items on disk queue         |
| ep_diskqueue_drain            | Total drained items on disk queue          |
| ep_diskqueue_pendingWrites    | Total bytes of pending writes              |


*** Active vBucket class stats

| Stat                          | Description                                |
|-------------------------------+--------------------------------------------|
| vb_active_num                 | Number of active vBuckets                  |
| vb_active_curr_items          | Number of in memory items                  |
| vb_active_num_non_resident    | Number of non-resident items               |
| vb_active_perc_mem_resident   | % memory resident                          |
| vb_active_eject               | Number of times item values got ejected    |
| vb_active_ht_memory           | Memory used to store keys and values       |
| vb_active_itm_memory          | Total item memory                          |
| vb_active_ops_create          | Number of create operations                |
| vb_active_ops_update          | Number of update operations                |
| vb_active_ops_delete          | Number of delete operations                |
| vb_active_ops_reject          | Number of rejected operations              |
| vb_active_queue_size          | Active items in disk queue                 |
| vb_active_queue_memory        | Memory used for disk queue                 |
| vb_active_queue_age           | Sum of disk queue item age in milliseconds |
| vb_active_queue_pending       | Total bytes of pending writes              |
| vb_active_queue_fill          | Total enqueued items                       |
| vb_active_queue_drain         | Total drained items                        |

*** Replica vBucket stats

| Stat                          | Description                                |
|-------------------------------+--------------------------------------------|
| vb_replica_num                | Number of replica vBuckets                 |
| vb_replica_curr_items         | Number of in memory items                  |
| vb_replica_num_non_resident   | Number of non-resident items               |
| vb_replica_perc_mem_resident  | % memory resident                          |
| vb_replica_eject              | Number of times item values got ejected    |
| vb_replica_ht_memory          | Memory used to store keys and values       |
| vb_replica_itm_memory         | Total item memory                          |
| vb_replica_ops_create         | Number of create operations                |
| vb_replica_ops_update         | Number of update operations                |
| vb_replica_ops_delete         | Number of delete operations                |
| vb_replica_ops_reject         | Number of rejected operations              |
| vb_replica_queue_size         | Replica items in disk queue                |
| vb_replica_queue_memory       | Memory used for disk queue                 |
| vb_replica_queue_age          | Sum of disk queue item age in milliseconds |
| vb_replica_queue_pending      | Total bytes of pending writes              |
| vb_replica_queue_fill         | Total enqueued items                       |
| vb_replica_queue_drain        | Total drained items                        |

*** Pending vBucket stats

| Stat                          | Description                                |
|-------------------------------+--------------------------------------------|
| vb_pending_num                | Number of pending vBuckets                 |
| vb_pending_curr_items         | Number of in memory items                  |
| vb_pending_num_non_resident   | Number of non-resident items               |
| vb_pending_perc_mem_resident  | % memory resident                          |
| vb_pending_eject              | Number of times item values got ejected    |
| vb_pending_ht_memory          | Memory used to store keys and values       |
| vb_pending_itm_memory         | Total item memory                          |
| vb_pending_ops_create         | Number of create operations                |
| vb_pending_ops_update         | Number of update operations                |
| vb_pending_ops_delete         | Number of delete operations                |
| vb_pending_ops_reject         | Number of rejected operations              |
| vb_pending_queue_size         | Pending items in disk queue                |
| vb_pending_queue_memory       | Memory used for disk queue                 |
| vb_pending_queue_age          | Sum of disk queue item age in milliseconds |
| vb_pending_queue_pending      | Total bytes of pending writes              |
| vb_pending_queue_fill         | Total enqueued items                       |
| vb_pending_queue_drain        | Total drained items                        |



** Tap stats

| ep_tap_total_queue       | Sum of tap queue sizes on the current      |
|                          | tap queues                                 |
| ep_tap_total_fetched     | Sum of all tap messages sent               |
| ep_tap_bg_max_pending    | The maximum number of bg jobs a tap        |
|                          | connection may have                        |
| ep_tap_bg_fetched        | Number of tap disk fetches                 |
| ep_tap_bg_fetch_requeued | Number of times a tap bg fetch task is     |
|                          | requeued.                                  |
| ep_tap_fg_fetched        | Number of tap memory fetches               |
| ep_tap_deletes           | Number of tap deletion messages sent       |
| ep_tap_throttled         | Number of tap messages refused due to      |
|                          | throttling.                                |
| ep_tap_keepalive         | How long to keep tap connection state      |
|                          | after client disconnect.                   |
| ep_tap_count             | Number of tap connections.                 |
| ep_tap_bg_num_samples    | The number of tap bg fetch samples         |
|                          | included in the avg                        |
| ep_tap_bg_min_wait       | The shortest time (µs) for a tap item      |
|                          | before it is serviced by the dispatcher    |
| ep_tap_bg_max_wait       | The longest time (µs) for a tap item       |
|                          | before it is serviced by the dispatcher    |
| ep_tap_bg_wait_avg       | The average wait time (µs) for a tap item  |
|                          | before it is serviced by the dispatcher    |
| ep_tap_bg_min_load       | The shortest time (µs) for a tap item to   |
|                          | be loaded from the persistence layer       |
| ep_tap_bg_max_load       | The longest time (µs) for a tap item to    |
|                          | be loaded from the persistence layer       |
| ep_tap_bg_load_avg       | The average time (µs) for a tap item to    |
|                          | be loaded from the persistence layer       |
| ep_tap_noop_interval     | The number of secs between a noop is added |
|                          | to an idle connection                      |
| ep_tap_backoff_period    | The number of seconds the tap connection   |
|                          | should back off after receiving ETMPFAIL   |
| ep_tap_queue_fill        | Total enqueued items                       |
| ep_tap_queue_drain       | Total drained items                        |
| ep_tap_queue_backoff     | Total back-off items                       |
| ep_tap_queue_backfill    | Number of backfill remaining               |
| ep_tap_queue_itemondisk  | Number of items remaining on disk          |


*** Per Tap Client Stats

Each stat begins with =ep_tapq:= followed by a unique /client_id/ and
another colon.  For example, if your client is named, =slave1=, the
=qlen= stat would be =ep_tapq:slave1:qlen=.

| type                      | The kind of tap connection (producer or  | PC |
|                           | consumer)                                |    |
| created                   | Creation time for the tap connection     | PC |
| supports_ack              | true if the connection use acks          | PC |
| connected                 | true if this client is connected         | PC |
| disconnects               | Number of disconnects from this client.  | PC |
| qlen                      | Queue size for the given client_id.      | P  |
| qlen_high_pri             | High priority tap queue items.           | P  |
| qlen_low_pri              | Low priority tap queue items.            | P  |
| vb_filters                | Size of connection vbucket filter set.   | P  |
| vb_filter                 | The content of the vbucket filter        | P  |
| rec_fetched               | Tap messages sent to the client.         | P  |
| rec_skipped               | Number of messages skipped due to        | P  |
|                           | tap reconnect with a different filter    | P  |
| idle                      | True if this connection is idle.         | P  |
| empty                     | True if this connection has no items.    | P  |
| complete                  | True if backfill is complete.            | P  |
| has_item                  | True when there is a bg fetched item     | P  |
|                           | ready.                                   | P  |
| has_queued_item           | True when there is a key ready to be     | P  |
|                           | looked up (may become fg or bg item)     | P  |
| bg_wait_for_result        | True if the max number of background     | P  |
|                           | operations is started                    | P  |
| bg_queue_size             | Number of bg fetches enqueued for this   | P  |
|                           | connection.                              | P  |
| bg_queued                 | Number of background fetches enqueued.   | P  |
| bg_result_size            | Number of ready background results.      | P  |
| bg_results                | Number of background results ready.      | P  |
| bg_jobs_issued            | Number of background jobs started.       | P  |
| bg_jobs_completed         | Number of background jobs completed.     | P  |
| bg_backlog_size           | Number of items pending bg fetch.        | P  |
| flags                     | Connection flags set by the client.      | P  |
| pending_disconnect        | true if we're hanging up on this client  | P  |
| paused                    | true if this client is blocked           | P  |
| pending_backfill          | true if we're still backfilling keys     | P  |
|                           | for this connection                      | P  |
| pending_disk_backfill     | true if we're still backfilling keys     | P  |
|                           | from disk for this connection            | P  |
| backfill_completed        | true if all items from backfill is       | P  |
|                           | successfully transmitted to the client   | P  |
| reconnects                | Number of reconnects from this client.   | P  |
| backfill_age              | The age of the start of the backfill.    | P  |
| ack_seqno                 | The current tap ACK sequence number.     | P  |
| recv_ack_seqno            | Last receive tap ACK sequence number.    | P  |
| ack_log_size              | Tap ACK backlog size.                    | P  |
| ack_window_full           | true if our tap ACK window is full.      | P  |
| expires                   | When this ACK backlog expires.           | P  |
| num_tap_nack              | The number of negative tap acks received | P  |
| num_tap_tmpfail_survivors | The number of items rescheduled due to   | P  |
|                           | a temporary nack.                        | P  |
| queue_memory              | Memory used for tap queue                | P  |
| queue_fill                | Total queued items                       | P  |
| queue_drain               | Total drained items                      | P  |
| queue_backoff             | Total back-off items                     | P  |
| queue_backfillremaining   | Number of backfill remaining             | P  |
| queue_itemondisk          | Number of items remaining on disk        | P  |
| total_backlog_size        | Num of remaining items for replication   | P  |
| num_delete                | Number of delete operations consumed     |  C |
| num_delete_failed         | Number of failed delete operations       |  C |
| num_flush                 | Number of flush operations               |  C |
| num_flush_failed          | Number of failed flush operations        |  C |
| num_mutation              | Number of mutation operations            |  C |
| num_mutation_failed       | Number of failed mutation operations     |  C |
| num_opaque                | Number of opaque operation consumed      |  C |
| num_opaque_failed         | Number of failed opaque operations       |  C |
| num_vbucket_set           | Number of vbucket set operations         |  C |
| num_vbucket_set_failed    | Number of failed vbucket set operations  |  C |
| num_unknown               | Number of unknown operations             |  C |

** Tap Aggregated Stats

Aggregated tap stats allow named tap connections to be logically
grouped and aggregated together by prefixes.

For example, if all of your tap connections started with =rebalance_=
or =replication_=, you could call =stats tapagg _= to request stats
grouped by everything before the first =_= character, giving you a set
for =rebalance= and a set for =replication=.

*** Results

| [prefix]:count              | Number of connections matching this prefix |
| [prefix]:qlen               | Total length of queues with this prefix    |
| [prefix]:backfill_remaining | Number of items needing to be backfilled   |
| [prefix]:backoff            | Total number of backoff events             |
| [prefix]:drain              | Total number of items drained              |
| [prefix]:fill               | Total number of items filled               |
| [prefix]:itemondisk         | Number of items remaining on disk          |
| [prefix]:total_backlog_size | Num of remaining items for replication     |

** Timing Stats

Timing stats provide histogram data from high resolution timers over
various operations within the system.

*** General Form

As this data is multi-dimensional, some parsing may be required for
machine processing.  It's somewhat human readable, but the =stats=
script mentioned in the Getting Started section above will do fancier
formatting for you.

Consider the following sample stats:

: STAT disk_insert_8,16 9488
: STAT disk_insert_16,32 290
: STAT disk_insert_32,64 73
: STAT disk_insert_64,128 86
: STAT disk_insert_128,256 48
: STAT disk_insert_256,512 2
: STAT disk_insert_512,1024 12
: STAT disk_insert_1024,2048 1

This tells you that =disk_insert= took 8-16µs 9,488 times, 16-32µs
290 times, and so on.

The same stats displayed through the =stats= CLI tool would look like
this:

: disk_insert (10008 total)
:    8us - 16us    : ( 94.80%) 9488 ###########################################
:    16us - 32us   : ( 97.70%)  290 #
:    32us - 64us   : ( 98.43%)   73
:    64us - 128us  : ( 99.29%)   86
:    128us - 256us : ( 99.77%)   48
:    256us - 512us : ( 99.79%)    2
:    512us - 1ms   : ( 99.91%)   12
:    1ms - 2ms     : ( 99.92%)    1


*** Available Stats

The following histograms are available from "timings" in the above
form to describe when time was spent doing various things:

| bg_wait               | bg fetches waiting in the dispatcher queue     |
| bg_load               | bg fetches waiting for disk                    |
| bg_tap_wait           | tap bg fetches waiting in the dispatcher queue |
| bg_tap_laod           | tap bg fetches waiting for disk                |
| pending_ops           | client connections blocked for operations      |
|                       | in pending vbuckets.                           |
| storage_age           | Analogous to ep_storage_age in main stats.     |
| data_age              | Analogous to ep_data_age in main stats.        |
| get_cmd               | servicing get requests                         |
| store_cmd             | servicing store requests                       |
| arith_cmd             | servicing incr/decr requests                   |
| get_vb_cmd            | servicing vbucket status requests              |
| set_vb_cmd            | servicing vbucket set state commands           |
| del_vb_cmd            | servicing vbucket deletion commands            |
| tap_vb_set            | servicing tap vbucket set state commands       |
| tap_vb_reset          | servicing tap vbucket reset commands           |
| tap_mutation          | servicing tap mutations                        |
| notify_io             | waking blocked connections                     |
| disk_insert           | waiting for disk to store a new item           |
| disk_update           | waiting for disk to modify an existing item    |
| disk_del              | waiting for disk to delete an item             |
| disk_vb_del           | waiting for disk to delete a vbucket           |
| disk_vb_chunk_del     | waiting for disk to delete a vbucket chunk     |
| disk_commit           | waiting for a commit after a batch of updates  |
| disk_invalid_item_del | Waiting for disk to delete a chunk of invalid  |
|                       | items with the old vbucket version             |

** Hash Stats

Hash stats provide information on your per-vbucket hash tables.

Requesting these stats does affect performance, so don't do it too
regularly, but it's useful for debugging certain types of performance
issues.  For example, if your hash table is tuned to have too few
buckets for the data load within it, the =max_depth= will be too large
and performance will suffer.

Each stat is prefixed with =vb_= followed by a number, a colon, then
the individual stat name.

For example, the stat representing the size of the hash table for
vbucket 0 is =vb_0:size=.

| state            | The current state of this vbucket                |
| size             | Number of hash buckets                           |
| locks            | Number of locks covering hash table operations   |
| min_depth        | Minimum number of items found in a bucket        |
| max_depth        | Maximum number of items found in a bucket        |
| reported         | Number of items this hash table reports having   |
| counted          | Number of items found while walking the table    |
| resized          | Number of times the hash table resized.          |
| mem_size         | Running sum of memory used by each item.         |
| mem_size_counted | Counted sum of current memory used by each item. |

** Checkpoint Stats

Checkpoint stats provide detailed information on per-vbucket checkpoint
datastructure.

Like Hash stats, requesting these stats has some impact on performance.
Therefore, please do not poll them from the server frequently.
Each stat is prefixed with =vb_= followed by a number, a colon, and then
each stat name.

| open_checkpoint_id   | ID of the current open checkpoint                   |
| num_tap_cursors      | Number of referencing TAP cursors                   |
| num_checkpoint_items | Number of total items in a checkpoint datastructure |
| num_checkpoints      | Number of checkpoints in a checkpoint datastructure |

* Details

** Ages

The difference between =ep_storage_age= and =ep_data_age= is somewhat
subtle, but when you consider that a given record may be updated
multiple times before hitting persistence, it starts to be clearer.

=ep_data_age= is how old the data we actually wrote is.

=ep_storage_age= is how long the object has been waiting to be
persisted.

** Too Young

=ep_too_young= is incremented every time an object is encountered
whose =data age= is more recent than is allowable for the persistence
layer.

For example, if an object that was queued five minutes ago is picked
off the =todo= queue and found to have been updated fifteen seconds
ago, it will not be stored, =ep_too_young= will be incremented, and
the key will go back on the input queue.

** Too Old

=ep_too_old= is incremented every time an object is encountered whose
=queue age= exceeds the =ep_queue_age_cap= setting.

=ep_queue_age_cap= generally exists as a safety net to prevent the
=ep_min_data_age= setting from preventing persistence altogether.

** Warming Up

Opening the data store is broken into three distinct phases:

*** Initializing

During the initialization phase, the server is not accepting
connections or otherwise functional.  This is often quick, but in a
server crash can take some time to perform recovery of the underlying
storage.

This time is made available via the =ep_dbinit= stat.

*** Warming Up

After initialization, warmup begins.  At this point, the server is
capable of taking new writes and responding to reads.  However, only
records that have been pulled out of the storage or have been updated
from other clients will be available for request.

(note that records read from persistence will not overwrite new
records captured from the network)

During this phase, =ep_warmup_thread= will report =running= and
=ep_warmed_up= will be increasing as records are being read.

*** Complete

Once complete, =ep_warmed_up= will stop increasing and
=ep_warmup_thread= will report =complete=.
