Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: adding a metric to capture event batch size DAT-909 #3070

Merged
merged 9 commits into from
Mar 15, 2024
3 changes: 2 additions & 1 deletion src/controllers/userTransform.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,10 @@
'(User transform - router:/customTransform ):: Request to transformer',
JSON.stringify(ctx.request.body),
);
const requestSize = Number(ctx.request.get('content-length'));

Check warning on line 18 in src/controllers/userTransform.ts

View check run for this annotation

Codecov / codecov/patch

src/controllers/userTransform.ts#L18

Added line #L18 was not covered by tests
const events = ctx.request.body as ProcessorTransformationRequest[];
const processedRespone: UserTransformationServiceResponse =
await UserTransformService.transformRoutine(events, ctx.state.features);
await UserTransformService.transformRoutine(events, ctx.state.features, requestSize);

Check warning on line 21 in src/controllers/userTransform.ts

View check run for this annotation

Codecov / codecov/patch

src/controllers/userTransform.ts#L21

Added line #L21 was not covered by tests
ctx.body = processedRespone.transformedEvents;
ControllerUtility.postProcess(ctx, processedRespone.retryStatus);
logger.debug(
Expand Down
14 changes: 9 additions & 5 deletions src/services/userTransform.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ import {
RetryRequestError,
extractStackTraceUptoLastSubstringMatch,
} from '../util/utils';
import { getMetadata, isNonFuncObject } from '../v0/util';
import { getMetadata, getTransformationMetadata, isNonFuncObject } from '../v0/util';
import { SUPPORTED_FUNC_NAMES } from '../util/ivmFactory';
import logger from '../logger';
import stats from '../util/stats';
Expand All @@ -28,6 +28,7 @@ export class UserTransformService {
public static async transformRoutine(
events: ProcessorTransformationRequest[],
features: FeatureFlags = {},
requestSize = 0,
): Promise<UserTransformationServiceResponse> {
let retryStatus = 200;
const groupedEvents: NonNullable<unknown> = groupBy(
Expand Down Expand Up @@ -162,16 +163,19 @@ export class UserTransformService {
),
);
stats.counter('user_transform_errors', eventsToProcess.length, {
transformationId: eventsToProcess[0]?.metadata?.transformationId,
workspaceId: eventsToProcess[0]?.metadata?.workspaceId,
status,
...metaTags,
...getTransformationMetadata(eventsToProcess[0]?.metadata),
});
} finally {
stats.timing('user_transform_request_latency', userFuncStartTime, {
workspaceId: eventsToProcess[0]?.metadata?.workspaceId,
transformationId: eventsToProcess[0]?.metadata?.transformationId,
...metaTags,
...getTransformationMetadata(eventsToProcess[0]?.metadata),
});

stats.histogram('user_transform_batch_size', requestSize, {
...metaTags,
...getTransformationMetadata(eventsToProcess[0]?.metadata),
});
}

Expand Down
20 changes: 20 additions & 0 deletions src/util/prometheus.js
Original file line number Diff line number Diff line change
Expand Up @@ -593,6 +593,10 @@ class Prometheus {
name: 'tp_batch_size',
help: 'Size of batch of events for tracking plan validation',
type: 'histogram',
buckets: [
1024, 102400, 524288, 1048576, 10485760, 20971520, 52428800, 104857600, 209715200,
524288000,
],
labelNames: [
'sourceType',
'destinationType',
Expand Down Expand Up @@ -670,6 +674,22 @@ class Prometheus {
'k8_namespace',
],
},
{
name: 'user_transform_batch_size',
help: 'user_transform_batch_size',
type: 'histogram',
labelNames: [
'workspaceId',
'transformationId',
'sourceType',
'destinationType',
'k8_namespace',
],
buckets: [
1024, 102400, 524288, 1048576, 10485760, 20971520, 52428800, 104857600, 209715200,
524288000,
], // 1KB, 100KB, 0.5MB, 1MB, 10MB, 20MB, 50MB, 100MB, 200MB, 500MB
},
{
name: 'source_transform_request_latency',
help: 'source_transform_request_latency',
Expand Down
Loading