pooler/README.org

• pooler - An OTP Process Pool Application
  • What pooler does
    • Protects the members of a pool from being used concurrently
    • Maintains the size of the pool
    • Manage multiple pools
  • Motivation
  • Usage and API
    • Pool Configuration
    • Using pooler
    • Metrics
    • Demo Quick Start
  • License

pooler - An OTP Process Pool Application

The pooler application allows you to manage pools of OTP behaviors such as gen_servers, gen_fsms, or supervisors, and provide consumers with exclusive access to pool members using pooler:take_member.

What pooler does

Protects the members of a pool from being used concurrently

The main pooler interface is pooler:take_member/0 and pooler:return_member/2. The pooler server will keep track of which members are in use and which are free. There is no need to call pooler:return_member if the consumer is a short-lived process; in this case, pooler will detect the consumer's normal exit and reclaim the member. To achieve this, pooler tracks the calling process of take_member as the consumer of the pool member. Thus pooler assumes that there is no middle-man process calling take_member and handing out the member pid to another worker process.

Maintains the size of the pool

You specify an initial and a maximum number of members in the pool. Pooler will create new members on demand until the maximum member count is reached. New pool members are added to replace member that crash. If a consumer crashes, the member it was using will be destroyed and replaced. Pooler will remove members that have not been used in cull_after minutes. Culling of members will not reduce a pool below the initial size.

Manage multiple pools

A common configuration is to have each pool contain client processes connected to a particular node in a cluster (think database read slaves). Pooler will randomly select a pool to fetch a member from. If the randomly selected pool has no free members, pooler will select a member from the pool with the most free members. If there is no pool with available members, pooler will return error_no_members.

Motivation

The need for pooler arose while writing an Erlang-based application that uses Riak for data storage. Riak's protocol buffer client is a gen_server process that initiates a connection to a Riak node. A pool is needed to avoid spinning up a new client for each request in the application. Reusing clients also has the benefit of keeping the vector clocks smaller since each client ID corresponds to an entry in the vector clock.

When using the Erlang protocol buffer client for Riak, one should avoid accessing a given client concurrently. This is because each client is associated with a unique client ID that corresponds to an element in an object's vector clock. Concurrent action from the same client ID defeats the vector clock. For some further explanation, see post 1 and post 2. Note that concurrent access to Riak's pb client is actual ok as long as you avoid updating the same key at the same time. So the pool needs to have checkout/checkin semantics that give consumers exclusive access to a client.

On top of that, in order to evenly load a Riak cluster and be able to continue in the face of Riak node failures, consumers should spread their requests across clients connected to each node. The client pool provides an easy way to load balance.

Usage and API

Pool Configuration

Pool configuration is specified in the pooler application's environment. This can be provided in a config file using -config or set at startup using application:set_env(pooler, pools, Pools). Here's an example config file that creates three pools of Riak pb clients each talking to a different node in a local cluster:

  % pooler.config
  % Start Erlang as: erl -config pooler
  % -*- mode: erlang -*-
  % pooler app config
  [
   {pooler, [
           {pools, [
                    [{name, "rc8081"},
                     {max_count, 5},
                     {init_count, 2},
                     {start_mfa,
                      {riakc_pb_socket, start_link, ["localhost", 8081]}}],
  
                    [{name, "rc8082"},
                     {max_count, 5},
                     {init_count, 2},
                     {start_mfa,
                      {riakc_pb_socket, start_link, ["localhost", 8082]}}],
  
                    [{name, "rc8083"},
                     {max_count, 5},
                     {init_count, 2},
                     {start_mfa,
                      {riakc_pb_socket, start_link, ["localhost", 8083]}}]
                   ]}
             %% if you want to enable metrics, set this to a module with
             %% an API conformant to the folsom_metrics module.
             %% If this config is missing, then no metrics are sent.
             %% {metrics_module, folsom_metrics}
          ]}
  ].

Each pool has a unique name, an initial and maximum number of members, and an {M, F, A} describing how to start members of the pool. When pooler starts, it will create members in each pool according to init_count.

Using pooler

Here's an example session:

application:start(pooler).
P = pooler:take_member(),
% use P
pooler:return_member(P, ok).

Once started, the main interaction you will have with pooler is through two functions, take_member/0 and return_member/2.

Call pooler:take_member() to obtain a member from a randomly selected pool. When you are done with it, return it to the pool using pooler:return_member(Pid, ok). If you encountered an error using the member, you can pass fail as the second argument. In this case, pooler will permanently remove that member from the pool and start a new member to replace it. If your process is short lived, you can omit the call to return_member. In this case, pooler will detect the normal exit of the consumer and reclaim the member.

Metrics

You can enable metrics collection by adding a metrics_module entry to pooler's app config. Metrics are disabled by default. The module specified must have an API matching that of the folsom_metrics module in folsom (to use folsom, specify {metrics_module, folsom_metrics}} and ensure that folsom is in your code path and has been started.

When enabled, the following metrics will be tracked:

Metric Label	Description
pooler.POOL_NAME.take_rate	meter recording rate at which take_member is called
pooler.error_no_members_count	counter indicating how many times take_member has returned error_no_members
pooler.killed_free_count	counter how many members have been killed when in the free state
pooler.killed_in_use_count	counter how many members have been killed when in the in_use state
pooler.event	history various error conditions

Demo Quick Start

1. Clone the repo:

git clone https://github.com/seth/pooler.git

1. Build and run tests:

cd pooler; make && make test

1. Start a demo

erl -pa .eunit ebin -config demo

Eshell V5.8.4  (abort with ^G)
1> application:start(pooler).
ok
2> M = pooler:take_member().
<0.49.0>
3> pooled_gs:get_id(M).
{"p2",#Ref<0.0.0.47>}
4> M2 = pooler:take_member().
<0.48.0>
5> pooled_gs:get_id(M2).
{"p2",#Ref<0.0.0.45>}
6> pooler:return_member(M).
ok
7> pooler:return_member(M2).
ok

License

Pooler is licensed under the Apache License Version 2.0. See the ./LICENSE file for details.