Generic Limiter to ensure N parallel operations


The limiter functionality is new. Please report any issues found on the retools Github issue tracker.

The limiter is useful when you want to make sure that only N operations for a given process happen at the same time, i.e.: concurrent requests to the same domain.

The limiter works by acquiring and releasing limits.

Creating a limiter:

from retools.limiter import Limiter

def do_something():
    limiter = Limiter(limit=10, prefix='my-operation')  # using default redis connection

    for i in range(100):
        if limiter.acquire_limit('operation-%d' % i):
            limiter.release_limit('operation-%d' % i)  # since we are releasing it synchronously
                                                       # all the 100 operations will be performed with
                                                       # one of them locked at a time

Specifying a default expiration in seconds:

def do_something():
    limiter = Limiter(limit=10, expiration_in_seconds=45)  # using default redis connection

Specifying a redis connection:

def do_something():
    limiter = Limiter(limit=10, redis=my_redis_connection)

Every time you try to acquire a limit, the expired limits you previously acquired get removed from the set.

This way if your process dies in the mid of its operation, the keys will eventually expire.

Public API Classes

class retools.limiter.Limiter(limit, redis=None, prefix='retools_limiter', expiration_in_seconds=10)[source]

Configures and limits operations

__init__(limit, redis=None, prefix='retools_limiter', expiration_in_seconds=10)[source]

Initializes a Limiter.

  • limit – An integer that describes the limit on the number of items
  • redis – A Redis instance. Defaults to the redis instance on the global_connection.
  • prefix – The default limit set name. Defaults to ‘retools_limiter’.
  • expiration_in_seconds – The number in seconds that keys should be locked if not explicitly released.
acquire_limit(key, expiration_in_seconds=None, retry=True)[source]

Tries to acquire a limit for a given key. Returns True if the limit can be acquired.

  • key – A string with the key to acquire the limit for. This key should be used when releasing.
  • expiration_in_seconds – The number in seconds that this key should be locked if not explicitly released. If this is not passed, the default is used.
  • key – Internal parameter that specifies if the operation should be retried. Defaults to True.

Releases a limit for a given key.

Parameters:key – A string with the key to release the limit on.