Skip to content

Commit

Permalink
readme update from douban#125
Browse files Browse the repository at this point in the history
  • Loading branch information
kentslaney committed Jun 6, 2024
1 parent fb03c5d commit 362921b
Showing 1 changed file with 56 additions and 45 deletions.
101 changes: 56 additions & 45 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5,15 +5,16 @@ libmc
|status| |pypiv| |pyversions| |wheel| |license|

libmc is a memcached client library for Python without any other
dependencies in runtime. It's mainly written in C++ and Cython. libmc
can be considered as a drop in replacement for libmemcached and
dependencies at runtime. It's mainly written in C++ and Cython and
can be considered a drop in replacement for libmemcached and
`python-libmemcached <https://github.com/douban/python-libmemcached>`__.

libmc is developing and maintaining by Douban Inc. Currently, It is
working in production environment, powering all web traffics in
douban.com. Realtime `benchmark
result <https://travis-ci.org/douban/libmc/builds/57124335#L1611>`__ is
available on travis.
libmc is developed and maintained by Douban Inc. Currently, it is
working in a production environment, powering all web traffic on
`douban.com <https://www.semrush.com/website/douban.com/overview/>`__. [#]_

.. [#] The link not an endorsement of the traffic estimate site's accuracy, just
a publicly available reference point
Build and Installation
----------------------
Expand All @@ -37,14 +38,14 @@ Usage:
Under the hood
--------------

Under the hood, libmc consists of 2 parts: an internal fully-functional
Under the hood, libmc consists of 2 parts: an internal, fully-functional
memcached client implementation in C++ and a Cython wrapper around that
implementation. Dynamic memory allocation and memory-copy are slow, so
we tried our best to avoid them. The ``set_multi`` command is not
natively supported by the `memcached
we've tried our best to avoid them. libmc also supports the ``set_multi``
command, which is not natively supported by the `memcached
protocol <https://github.com/memcached/memcached/blob/master/doc/protocol.txt>`__.
Some techniques are applied to make ``set_multi`` command extremely fast
in libmc (compared to some other similiar libraries).
Some techniques have been applied to make ``set_multi`` command extremely fast
in libmc (compared to similiar libraries).

Configuration
-------------
Expand Down Expand Up @@ -77,24 +78,23 @@ Configuration
mc.config(MC_RETRY_TIMEOUT, 5) # 5 s
- ``servers``: is a list of memcached server addresses. Each address
can be in format of ``hostname[:port] [alias]``. ``port`` and ``alias``
are optional. If ``port`` is not given, default port ``11211`` will
be used. ``alias`` will be used to compute server hash if given,
otherwise server hash will be computed based on ``host`` and ``port``
(i.e.: If ``port`` is not given or it is equal to ``11211``, ``host``
will be used to compute server hash. If ``port`` is not equal to ``11211``,
``host:port`` will be used).
- ``do_split``: Memcached server will refuse to store value if size >=
1MB, if ``do_split`` is enabled, large value (< 10 MB) will be
splitted into several blocks. If the value is too large (>= 10 MB),
it will not be stored. default: ``True``
- ``comp_threshold``: All kinds of values will be encoded into string
buffer. If ``buffer length > comp_threshold > 0``, it will be
compressed using zlib. If ``comp_threshold = 0``, string buffer will
never be compressed using zlib. default: ``0``
- ``noreply``: Whether to enable memcached's ``noreply`` behaviour.
default: ``False``
- ``servers``: a list of memcached server addresses. Each address
should be formated as ``hostname[:port] [alias]``, where ``port`` and
``alias`` are optional. If ``port`` is not given, the default port ``11211``
will be used. If given, ``alias`` will be used to compute the server hash,
which would otherwise be computed based on ``host`` and ``port``
(i.e. whichever portion is given).
- ``do_split``: splits large values (up to 10MB) into chunks (<1MB). The
memcached server implementation will not store items larger than 1MB,
however in some environments it is beneficial to shard up to 10MB of data.
Attempts to store more than that are ignored. Default: ``True``.
- ``comp_threshold``: compresses large values using zlib. If
``buffer length > comp_threshold > 0`` (in bytes), the buffer will be
compressed. If ``comp_threshold == 0``, the string buffer will never be
compressed. Default: ``0``
- ``noreply``: controls memcached's
``noreply`` `feature <https://github.com/memcached/memcached/wiki/CommonFeatures#noreplyquiet>`__.
Default: ``False``
- ``prefix``: The key prefix. default: ``''``
- ``hash_fn``: hashing function for keys. possible values:

Expand All @@ -110,16 +110,16 @@ Configuration
implementions in libmemcached.

- ``failover``: Whether to failover to next server when current server
is not available. default: ``False``
is not available. Default: ``False``

- ``MC_POLL_TIMEOUT`` Timeout parameter used during set/get procedure.
(default: ``300`` ms)
Default: ``300`` ms
- ``MC_CONNECT_TIMEOUT`` Timeout parameter used when connecting to
memcached server on initial phase. (default: ``100`` ms)
- ``MC_RETRY_TIMEOUT`` When a server is not available dur to server-end
error. libmc will try to establish the broken connection in every
``MC_RETRY_TIMEOUT`` s until the connection is back to live.(default:
``5`` s)
memcached server in the initial phase. Default: ``100`` ms
- ``MC_RETRY_TIMEOUT`` When a server is not available due to server-end
error, libmc will try to establish the broken connection in every
``MC_RETRY_TIMEOUT`` s until the connection is back to live. Default:
``5`` s

**NOTE:** The hashing algorithm for host mapping on continuum is always
md5.
Expand All @@ -140,7 +140,7 @@ FAQ
Does libmc support PHP?
^^^^^^^^^^^^^^^^^^^^^^^

No. But if you like, you can write a wrapper for PHP based on the C++
No, but, if you like, you can write a wrapper for PHP based on the C++
implementation.

Is Memcached binary protocol supported ?
Expand All @@ -151,7 +151,7 @@ No. Only Memcached ASCII protocol is supported currently.
Why reinventing the wheel?
^^^^^^^^^^^^^^^^^^^^^^^^^^

Before libmc, we're using
Before libmc, we were using
`python-libmemcached <https://github.com/douban/python-libmemcached>`__,
which is a python extention for
`libmemcached <http://libmemcached.org/libMemcached.html>`__.
Expand All @@ -161,21 +161,32 @@ still some unsolved bugs.
Is libmc thread-safe ?
^^^^^^^^^^^^^^^^^^^^^^

libmc is a single-threaded memcached client. If you initialize a libmc
client in one thread but reuse that in another thread, a Python
Exception ``ThreadUnsafe`` will raise in Python.
Yes. ``libmc.ThreadedClient`` is a thread-safe client implementation. To hold
access for more than one request, ``libmc.ClientPool`` can be used with Python
``with`` statements. ``libmc.Client``, however, is a single-threaded memcached
client. If you initialize a standard client in one thread but reuse that in
another thread, a Python ``ThreadUnsafe`` Exception will be raised.

Is libmc compatible with gevent?
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Yes, with the help of `greenify <https://github.com/douban/greenify>`__,
libmc is friendly to gevent. Read ``tests/shabby/gevent_issue.py`` for
details.
details. ``libmc.ThreadedClient`` and ``libmc.ClientPool`` are not compatible.
[#]_

.. [#] In order to use a single executable for multiple greenlet contexts,
gevent has to `copy thread memory
<https://github.com/python-greenlet/greenlet/blob/937f150e07823ee03344aeeb5111c0bb371a831d/src/greenlet/greenlet.cpp#L105>`__
to and from the same stack space. This doesn't affect Python references,
which are handed off through gevent, but makes it impossible for shared
libraries to pass memory addresses across greenlets, which is required for
the worker pool.
**Notice:**

`gevent.monkey.patch_all()` will override
`threading.current_thread().ident` to Greenlet's ID,
``gevent.monkey.patch_all()`` will override
``threading.current_thread().ident`` to Greenlet's ID,
this will cause libmc to throw a ThreadUnSafe error
or run into dead lock, you should only patch the things
that you need, e.g.
Expand Down

0 comments on commit 362921b

Please sign in to comment.