Skip to content

Commit

Permalink
Redis - v4 using node redis library (#1198)
Browse files Browse the repository at this point in the history 
* redis - v4 - using node-redis library

* initial check in starting fresh

* adding tests for constructor

* adding in set, delete, and get

* adding in namespace support

* fixing clear on namespace

* adding has method

* adding in more tests

* adding in keyv test-suite

* adding in jsDoc

* fixing iteration tests

* adding in useUnlink

* adding in clustering and tls support

* adding in createKeyv

* adding in Keyv exports

* Update index.ts

* adding in the table of contents

* Update README.md

* Update README.md
  • Loading branch information
@jaredwray
jaredwray authored Nov 12, 2024
1 parent 3ad0bd5 commit 359a20f
Show file tree
Hide file tree
Showing 6 changed files with 989 additions and 323 deletions.
219 changes: 178 additions & 41 deletions packages/redis/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,93 +9,230 @@

Redis storage adapter for [Keyv](https://github.com/jaredwray/keyv).

TTL functionality is handled directly by Redis so no timestamps are stored and expired keys are cleaned up internally.
# Features
* Built on top of [redis](https://npmjs.com/package/redis).
* TTL is handled directly by Redis.
* Supports Redis Clusters.
* Url connection string support or pass in your Redis Options
* Easily add in your own Redis client.
* Namespace support for key management.
* Unlink as default delete method for performance.
* Access to the Redis client for advanced use cases.
* Keyv and Redis Libraries are exported for advanced use cases.
* `createKeyv` function for easy creation of Keyv instances.
* jsDoc comments for easy documentation.
* CJS / ESM and TypeScript supported out of the box.

# Table of Contents
* [Usage](#usage)
* [Namespaces](#namespaces)
* [Performance Considerations](#performance-considerations)
* [High Memory Usage on Redis Server](#high-memory-usage-on-redis-server)
* [Using Cacheable with Redis](#using-cacheable-with-redis)
* [Clustering and TLS Support](#clustering-and-tls-support)
* [API](#api)
* [Migrating from v3 to v4](#migrating-from-v3-to-v4)
* [About Redis Sets and its Support in v4](#about-redis-sets-and-its-support-in-v4)
* [License](#license)

# Usage

Here is a standard use case where we implement `Keyv` and `@keyv/redis`:

## Install
```js
import Keyv from 'keyv';
import KeyvRedis from '@keyv/redis';

```shell
npm install --save keyv @keyv/redis
const keyv = new Keyv(new KeyvRedis('redis://user:pass@localhost:6379'));
keyv.on('error', handleConnectionError);
```

## Usage
Here you can pass in the Redis options directly:

```js
import Keyv from 'keyv';
import KeyvRedis from '@keyv/redis';

const keyv = new Keyv(new KeyvRedis('redis://user:pass@localhost:6379'));
keyv.on('error', handleConnectionError);
```
const redisOptions = {
url: 'redis://localhost:6379', // The Redis server URL (use 'rediss' for TLS)
password: 'your_password', // Optional password if Redis has authentication enabled

Any valid [`Redis`](https://github.com/luin/ioredis#connect-to-redis) options will be passed directly through.
socket: {
host: 'localhost', // Hostname of the Redis server
port: 6379, // Port number
reconnectStrategy: (retries) => Math.min(retries * 50, 2000), // Custom reconnect logic

e.g:
tls: false, // Enable TLS if you need to connect over SSL
keepAlive: 30000, // Keep-alive timeout (in milliseconds)
}
};

```js
const keyv = new Keyv(new KeyvRedis('redis://user:pass@localhost:6379', { disable_resubscribing: true }));
const keyv = new Keyv(new KeyvRedis(redisOptions));
```

Or you can manually create a storage adapter instance and pass it to Keyv:
Or you can create a new Redis instance and pass it in with `KeyvOptions`:

```js
import Keyv from 'keyv';
import KeyvRedis from '@keyv/redis';
import KeyvRedis, { createClient } from '@keyv/redis';

const keyvRedis = new KeyvRedis('redis://user:pass@localhost:6379');
const redis = createClient('redis://user:pass@localhost:6379', { namespace: 'my-namespace'});
const keyvRedis = new KeyvRedis(redis);
const keyv = new Keyv({ store: keyvRedis });
```

Or reuse a previous Redis instance:
Here is the same example but with the `Keyv` instance created with the `createKeyv` function:

```js
import Keyv from 'keyv';
import Redis from 'ioredis';
import KeyvRedis from '@keyv/redis';
import { createKeyv } from '@keyv/redis';

const redis = new Redis('redis://user:pass@localhost:6379');
const keyvRedis = new KeyvRedis(redis);
const keyv = new Keyv({ store: keyvRedis });
const keyv = createKeyv('redis://user:pass@localhost:6379', { namespace: 'my-namespace' });
```

Or reuse a previous Redis cluster:
You only have to import the `@keyv/redis` library if you are using the `createKeyv` function. 🎉 Otherwise, you can import `Keyv` and `@keyv/redis` independently.

# Namspaces

You can set a namespace for your keys. This is useful if you want to manage your keys in a more organized way. Here is an example of how to set a namespace:

```js
import Keyv from 'keyv';
import Redis from 'ioredis';
import KeyvRedis from '@keyv/redis';

const redis = new Redis.Cluster('redis://user:pass@localhost:6379');
const keyvRedis = new KeyvRedis(redis);
const keyv = new Keyv({ store: keyvRedis });
const keyv = new Keyv(new KeyvRedis('redis://user:pass@localhost:6379', { namespace: 'my-namespace' }));
```
## Options

### useRedisSets
This will prefix all keys with `my-namespace:`. You can also set the namespace after the fact:

```js
keyv.namespace = 'my-namespace';
```

NOTE: If you plan to do many clears or deletes, it is recommended to read the [Performance Considerations](#performance-considerations) section.

# Performance Considerations

With namespaces being prefix based it is critical to understand some of the performance considerations we have made:
* `clear()` - We use the `SCAN` command to iterate over keys. This is a non-blocking command that is more efficient than `KEYS`. In addition we are using `UNLINK` by default instead of `DEL`. Even with that if you are iterating over a large dataset it can still be slow. It is highly recommended to use the `namespace` option to limit the keys that are being cleared and if possible to not use the `clear()` method in high performance environments.

The `useRedisSets` option lets you decide whether to use Redis sets for key management. By default, this option is set to `true`.
* `delete()` - By default we are now using `UNLINK` instead of `DEL` for deleting keys. This is a non-blocking command that is more efficient than `DEL`. If you are deleting a large number of keys it is recommended to use the `deleteMany()` method instead of `delete()`.

When `useRedisSets` is enabled (`true`):
* `clearBatchSize` - The `clearBatchSize` option is set to `1000` by default. This is because Redis has a limit of 1000 keys that can be deleted in a single batch.

- A namespace for the Redis sets is created, and all created keys are added to this. This allows for group management of keys.
- When a key is deleted, it's removed not only from the main storage but also from the Redis set.
- When clearing all keys (using the `clear` function), all keys in the Redis set are looked up for deletion. The set itself is also deleted.
* `useUnlink` - This option is set to `true` by default. This is because `UNLINK` is a non-blocking command that is more efficient than `DEL`. If you are not using `UNLINK` and are doing a lot of deletes it is recommended to set this option to `true`.

**Note**: In high-performance scenarios, enabling `useRedisSets` might lead to memory leaks. If you're running a high-performance application or service, it is recommended to set `useRedisSets` to `false`.
* `setMany`, `getMany`, `deleteMany` - These methods are more efficient than their singular counterparts. If you are doing multiple operations it is recommended to use these methods.

If you decide to set `useRedisSets` as `false`, keys will be handled individually and Redis sets won't be utilized.
If you want to see even better performance please see the [Using Cacheable with Redis](#using-cacheable-with-redis) section as it has non-blocking and in-memory primary caching that goes along well with this library and Keyv.

However, please note that setting `useRedisSets` to `false` could lead to performance issues in production when using the `clear` function, as it will need to iterate over all keys to delete them.
# High Memory Usage on Redis Server

#### Example
This is because we are using `UNLINK` by default instead of `DEL`. This is a non-blocking command that is more efficient than `DEL` but will slowly remove the memory allocation.

Here's how you can use the `useRedisSets` option:
If you are deleting or clearing a large number of keys you can disable this by setting the `useUnlink` option to `false`. This will use `DEL` instead of `UNLINK` and should reduce the memory usage.

```js
const keyv = new Keyv(new KeyvRedis('redis://user:pass@localhost:6379', { useUnlink: false }));
// Or
keyv.useUnlink = false;
```

# Using Cacheable with Redis

If you are wanting to see even better performance with Redis, you can use [Cacheable](https://npmjs.org/package/cacheable) which is a multi-layered cache library that has in-memory primary caching and non-blocking secondary caching. Here is an example of how to use it with Redis:

```js
import KeyvRedis from '@keyv/redis';
import Cacheable from 'cacheable';

const secondary = new KeyvRedis('redis://user:pass@localhost:6379');

const cache = new Cacheable( { secondary } );
```

For even higher performance you can set the `nonBlocking` option to `true`:

```js
const cache = new Cacheable( { secondary, nonBlocking: true } );
```

This will make it so that the secondary does not block the primary cache and will be very fast. 🚀

# Clustering and TLS Support

If you are using a Redis Cluster or need to use TLS, you can pass in the `redisOptions` directly. Here is an example of how to do that:

```js
import Keyv from 'keyv';
import KeyvRedis, { createCluster } from '@keyv/redis';

const cluster = createCluster({
rootNodes: [
{
url: 'redis://127.0.0.1:7000',
},
{
url: 'redis://127.0.0.1:7001',
},
{
url: 'redis://127.0.0.1:7002',
},
],
});

const keyv = new Keyv({ store: new KeyvRedis(cluster) });
```

Here is an example of how to use TLS:

```js
import Keyv from 'keyv';
import KeyvRedis from '@keyv/redis';

const keyv = new Keyv(new KeyvRedis('redis://user:pass@localhost:6379', { useRedisSets: false }));
const tlsOptions = {
socket: {
host: 'localhost',
port: 6379,
tls: true, // Enable TLS connection
rejectUnauthorized: false, // Ignore self-signed certificate errors (for testing)

// Alternatively, provide CA, key, and cert for mutual authentication
ca: fs.readFileSync('/path/to/ca-cert.pem'),
cert: fs.readFileSync('/path/to/client-cert.pem'), // Optional for client auth
key: fs.readFileSync('/path/to/client-key.pem'), // Optional for client auth
}
};

const keyv = new Keyv({ store: new KeyvRedis(tlsOptions) });
```

## License
# API
* **constructor([connection], [options])**
* **namespace** - The namespace to use for the keys.
* **client** - The Redis client instance.
* **keyPrefixSeparator** - The separator to use between the namespace and key.
* **clearBatchSize** - The number of keys to delete in a single batch.
* **useUnlink** - Use the `UNLINK` command for deleting keys isntead of `DEL`.
* **set** - Set a key.
* **setMany** - Set multiple keys.
* **get** - Get a key.
* **getMany** - Get multiple keys.
* **has** - Check if a key exists.
* **hasMany** - Check if multiple keys exist.
* **delete** - Delete a key.
* **deleteMany** - Delete multiple keys.
* **clear** - Clear all keys. If the `namespace` is set it will only clear keys with that namespace.
* **disconnect** - Disconnect from the Redis server.
* **iterator** - Create a new iterator for the keys.

# Migrating from v3 to v4

The main change in v4 is the removal of the `ioredis` library in favor of the `@keyv/redis` library. This was done to provide a more consistent experience across all Keyv storage adapters. The `@keyv/redis` library is a wrapper around the `redis` library and provides a more consistent experience across all Keyv storage adapters. The only other change is that we no longer do redis sets as they caused performance issues.

# About Redis Sets and its Support in v4

We no longer support redis sets. This is due to the fact that it caused significant performance issues and was not a good fit for the library.

# License

[MIT © Jared Wray](LISCENCE)
6 changes: 3 additions & 3 deletions packages/redis/package.json
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@keyv/redis",
"version": "3.0.1",
"version": "4.0.0",
"description": "Redis storage adapter for Keyv",
"type": "module",
"main": "dist/index.cjs",
Expand Down Expand Up @@ -58,11 +58,11 @@
},
"homepage": "https://github.com/jaredwray/keyv",
"dependencies": {
"ioredis": "^5.4.1"
"redis": "^4.7.0",
"keyv": "*"
},
"devDependencies": {
"@keyv/test-suite": "*",
"keyv": "*",
"rimraf": "^6.0.1",
"timekeeper": "^2.3.1",
"tsd": "^0.31.2",
Expand Down
Loading

0 comments on commit 359a20f

Please sign in to comment.