feat: Add Cloudflare AI Gateway support #821

w3-bounty · 2024-12-03T12:16:09Z

Cloudflare AI Gateway Integration

Relates to

Enhances model provider integration by adding support for Cloudflare AI Gateway

Risks

Low - Changes are isolated to model provider routing and include fallback mechanisms:

If gateway configuration is incomplete/invalid, falls back to direct API calls
Comprehensive logging added for debugging
No changes to core model interaction logic

Background

What does this PR do?

Adds support for routing API calls through Cloudflare AI Gateway, providing:

Improved latency through Cloudflare's global network
Detailed analytics and monitoring
Cost optimization through request caching
Enhanced security and access control

Changes Made

packages/core/src/generation.ts:
- Added getCloudflareGatewayBaseURL utility function for consistent gateway URL construction
- Implemented gateway support for OpenAI, Anthropic, and Groq providers
- Added comprehensive logging for debugging
- Implemented fallback logic when gateway is disabled/misconfigured
docs/docs/guides/configuration.md:
- Added new section documenting Cloudflare AI Gateway integration
- Included configuration instructions and environment variables
- Listed supported providers and benefits
- Explained URL format and fallback behavior

.env.example:

Added Cloudflare Gateway configuration variables:

CLOUDFLARE_GW_ENABLED=true
CLOUDFLARE_AI_ACCOUNT_ID=your-account-id
CLOUDFLARE_AI_GATEWAY_ID=your-gateway-id

What kind of change is this?

Feature (non-breaking change which adds functionality)

Documentation changes

Documentation has been updated in configuration.md with:

Gateway setup instructions
Configuration options
Benefits and supported providers

Testing

Where should a reviewer start?

Review the getCloudflareGatewayBaseURL function in generation.ts
Check gateway integration in provider handlers
Verify documentation clarity in configuration.md

Detailed testing steps

Configure Cloudflare AI Gateway:

CLOUDFLARE_GW_ENABLED=true
CLOUDFLARE_AI_ACCOUNT_ID=your-account-id
CLOUDFLARE_AI_GATEWAY_ID=your-gateway-id

Test with different providers:
- Verify OpenAI requests route through gateway
- Verify Anthropic requests route through gateway
- Verify Groq requests route through gateway
Test fallback behavior:
- Disable gateway and verify direct API calls
- Use invalid configuration and verify fallback
- Check logging messages for debugging info

Expected Results

When gateway is properly configured:
- Requests route through Cloudflare
- Logs show gateway URL being used
- Response handling remains unchanged
When gateway is disabled/misconfigured:
- Fallback to direct API calls
- Warning logs indicate configuration issues
- Service continues to function normally

Discord username

w3_bounty

- Add Cloudflare Gateway integration for OpenAI, Anthropic, and Groq - Implement gateway URL construction and fallback logic - Add comprehensive logging for debugging - Update configuration documentation with gateway setup instructions

odilitime

seems to force all openai like apis to use CF, I think it needs to be more optional. I love the utility, desperately needed but we need fallback to the old URLs before we can merge thie

lostgirldev · 2024-12-30T20:22:23Z

.env.example

+CLOUDFLARE_GW_ENABLED=        # Set to true to enable Cloudflare AI Gateway
+CLOUDFLARE_AI_ACCOUNT_ID=     # Cloudflare AI Account ID - found in the Cloudflare Dashboard under AI Gateway
+CLOUDFLARE_AI_GATEWAY_ID=     # Cloudflare AI Gateway ID - found in the Cloudflare Dashboard under AI Gateway


@w3-bounty The addition of sensitive configuration variables such as CLOUDFLARE_AI_ACCOUNT_ID and CLOUDFLARE_AI_GATEWAY_ID in the example environment file could lead to security vulnerabilities if this file is exposed. Consider using placeholders or ensuring that sensitive information is not included in example files. For instance, use CLOUDFLARE_AI_ACCOUNT_ID=YOUR_ACCOUNT_ID instead.

lostgirldev · 2024-12-30T20:22:42Z

packages/core/src/generation.ts

+function getCloudflareGatewayBaseURL(runtime: IAgentRuntime, provider: string): string | undefined {
+    const isCloudflareEnabled = runtime.getSetting("CLOUDFLARE_GW_ENABLED") === "true";
+    const cloudflareAccountId = runtime.getSetting("CLOUDFLARE_AI_ACCOUNT_ID");
+    const cloudflareGatewayId = runtime.getSetting("CLOUDFLARE_AI_GATEWAY_ID");
+
+    elizaLogger.debug("Cloudflare Gateway Configuration:", {
+        isEnabled: isCloudflareEnabled,
+        hasAccountId: !!cloudflareAccountId,
+        hasGatewayId: !!cloudflareGatewayId,
+        provider: provider
+    });
+
+    if (!isCloudflareEnabled) {
+        elizaLogger.debug("Cloudflare Gateway is not enabled");
+        return undefined;
+    }
+
+    if (!cloudflareAccountId) {
+        elizaLogger.warn("Cloudflare Gateway is enabled but CLOUDFLARE_AI_ACCOUNT_ID is not set");
+        return undefined;
+    }
+
+    if (!cloudflareGatewayId) {
+        elizaLogger.warn("Cloudflare Gateway is enabled but CLOUDFLARE_AI_GATEWAY_ID is not set");
+        return undefined;
+    }
+
+    const baseURL = `https://gateway.ai.cloudflare.com/v1/${cloudflareAccountId}/${cloudflareGatewayId}/${provider.toLowerCase()}`;
+    elizaLogger.info("Using Cloudflare Gateway:", {
+        provider,
+        baseURL,
+        accountId: cloudflareAccountId,
+        gatewayId: cloudflareGatewayId
+    });
+
+    return baseURL;
+}


@w3-bounty The function getCloudflareGatewayBaseURL does not handle potential errors when retrieving settings from runtime. If the runtime is not properly configured or if there are issues accessing the settings, it could lead to unexpected behavior. Consider adding error handling to manage these scenarios gracefully.

lostgirldev · 2024-12-30T20:22:42Z

packages/core/src/generation.ts

+                const baseURL = getCloudflareGatewayBaseURL(runtime, 'openai') || endpoint;
+                elizaLogger.debug("OpenAI baseURL result:", { baseURL });
+
+                const openai = createOpenAI({ apiKey, baseURL });


@w3-bounty The baseURL is derived from getCloudflareGatewayBaseURL, which can return undefined. If this happens, the createOpenAI function will be called with an undefined baseURL, potentially leading to runtime errors. Ensure that a valid baseURL is always provided or handle the case where it is undefined.

lostgirldev · 2024-12-30T20:22:42Z

packages/core/src/generation.ts

+                const baseURL = getCloudflareGatewayBaseURL(runtime, 'anthropic');
+                elizaLogger.debug("Anthropic baseURL result:", { baseURL });

+                const anthropic = createAnthropic({ apiKey, baseURL });


@w3-bounty Similar to the previous comment, the baseURL for the createAnthropic function can also be undefined. This could lead to issues when the function is called. Implement a check to ensure that baseURL is valid before proceeding.

lostgirldev · 2024-12-30T20:22:42Z

packages/core/src/generation.ts

+                const baseURL = getCloudflareGatewayBaseURL(runtime, 'groq');
+                elizaLogger.debug("Groq baseURL result:", { baseURL });

+                const groq = createGroq({ apiKey, baseURL });


@w3-bounty Again, the baseURL for the createGroq function can be undefined. This inconsistency in handling undefined values across different model initializations can lead to bugs. Ensure that a valid baseURL is always passed to the create functions.

feat: Add Cloudflare AI Gateway support

e0a7aa4

- Add Cloudflare Gateway integration for OpenAI, Anthropic, and Groq - Implement gateway URL construction and fallback logic - Add comprehensive logging for debugging - Update configuration documentation with gateway setup instructions

shakkernerd added the Discussion label Dec 3, 2024

odilitime requested changes Dec 10, 2024

View reviewed changes

lalalune changed the base branch from main to develop December 14, 2024 09:36

odilitime deleted the branch elizaOS:develop December 17, 2024 02:33

odilitime closed this Dec 17, 2024

odilitime reopened this Dec 17, 2024

shakkernerd deleted the branch elizaOS:develop December 17, 2024 03:45

shakkernerd closed this Dec 17, 2024

odilitime reopened this Dec 17, 2024

shakkernerd deleted the branch elizaOS:develop December 22, 2024 07:01

shakkernerd closed this Dec 22, 2024

odilitime reopened this Dec 22, 2024

lostgirldev reviewed Dec 30, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add Cloudflare AI Gateway support #821

feat: Add Cloudflare AI Gateway support #821

w3-bounty commented Dec 3, 2024

odilitime left a comment

lostgirldev Dec 30, 2024

lostgirldev Dec 30, 2024

lostgirldev Dec 30, 2024

lostgirldev Dec 30, 2024

lostgirldev Dec 30, 2024

feat: Add Cloudflare AI Gateway support #821

Are you sure you want to change the base?

feat: Add Cloudflare AI Gateway support #821

Conversation

w3-bounty commented Dec 3, 2024

Cloudflare AI Gateway Integration

Relates to

Risks

Background

What does this PR do?

Changes Made

What kind of change is this?

Documentation changes

Testing

Where should a reviewer start?

Detailed testing steps

Expected Results

Discord username

odilitime left a comment

Choose a reason for hiding this comment

lostgirldev Dec 30, 2024

Choose a reason for hiding this comment

lostgirldev Dec 30, 2024

Choose a reason for hiding this comment

lostgirldev Dec 30, 2024

Choose a reason for hiding this comment

lostgirldev Dec 30, 2024

Choose a reason for hiding this comment

lostgirldev Dec 30, 2024

Choose a reason for hiding this comment