Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Online Scoring] Automation Rules endpoints #945

Open
wants to merge 10 commits into
base: main
Choose a base branch
from
Open

[Online Scoring] Automation Rules endpoints #945

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
1315442
## Details
ldaugusto Dec 20, 2024
59cb8c2
cleaning up
ldaugusto Dec 20, 2024
bdf11fe
Splitting Rule and Evaluator types
ldaugusto Dec 20, 2024
6d731f6
cleaning up
ldaugusto Dec 21, 2024
eccaa4b
moving 'sampling rate' to the base rule; other rule actions would als…
ldaugusto Dec 21, 2024
a79b400
removing RequestContext references from Service
ldaugusto Dec 21, 2024
a20a452
Read-only rule evaluator ids
ldaugusto Dec 21, 2024
a544940
cleaning up
ldaugusto Dec 21, 2024
84f5197
moving project to endpoint root and cleaning up descriptions
ldaugusto Dec 23, 2024
f4310cb
Removing hard coded 'evaluator' from sqls to Action bind. Defensive p…
ldaugusto Dec 23, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 55 additions & 0 deletions apps/opik-backend/src/main/java/com/comet/opik/api/AutomationRuleEvaluator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
package com.comet.opik.api;

import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import com.fasterxml.jackson.annotation.JsonView;
import com.fasterxml.jackson.databind.PropertyNamingStrategies;
import com.fasterxml.jackson.databind.annotation.JsonNaming;
import io.swagger.v3.oas.annotations.media.Schema;
import jakarta.validation.constraints.Pattern;
import lombok.Builder;

import java.time.Instant;
import java.util.List;
import java.util.UUID;

import static com.comet.opik.utils.ValidationUtils.NULL_OR_NOT_BLANK;

@Builder(toBuilder = true)
@JsonIgnoreProperties(ignoreUnknown = true)
@JsonNaming(PropertyNamingStrategies.SnakeCaseStrategy.class)
public record AutomationRuleEvaluator(
// Fields and methods
@JsonView({View.Public.class, View.Write.class}) UUID id,
ldaugusto marked this conversation as resolved.
Show resolved Hide resolved
@JsonView({View.Public.class, View.Write.class}) UUID projectId,
@JsonView({View.Public.class, View.Write.class}) AutomationRuleEvaluatorType evaluatorType,
ldaugusto marked this conversation as resolved.
Show resolved Hide resolved

@JsonView({View.Public.class,
View.Write.class}) @Pattern(regexp = NULL_OR_NOT_BLANK, message = "must not be blank") String code,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This field should probably be a polymorphic model, so we can define LLM as a judge evaluators and python evaluators, with different payloads.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's update code to be a JsonNode.


@JsonView({View.Public.class, View.Write.class}) float samplingRate,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is validated at the DAO level, please add validations on this API object as well.


@JsonView({View.Public.class}) @Schema(accessMode = Schema.AccessMode.READ_ONLY) Instant createdAt,
@JsonView({View.Public.class}) @Schema(accessMode = Schema.AccessMode.READ_ONLY) String createdBy,
@JsonView({View.Public.class}) @Schema(accessMode = Schema.AccessMode.READ_ONLY) Instant lastUpdatedAt,
@JsonView({View.Public.class}) @Schema(accessMode = Schema.AccessMode.READ_ONLY) String lastUpdatedBy){

public static class View {
public static class Write {}
public static class Public {}
}

@Builder(toBuilder = true)
public record AutomationRuleEvaluatorPage(
@JsonView( {
View.Public.class}) int page,
@JsonView({View.Public.class}) int size,
@JsonView({View.Public.class}) long total,
@JsonView({View.Public.class}) List<AutomationRuleEvaluator> content)
implements
Page<AutomationRuleEvaluator>{

public static AutomationRuleEvaluator.AutomationRuleEvaluatorPage empty(int page) {
return new AutomationRuleEvaluator.AutomationRuleEvaluatorPage(page, 0, 0, List.of());
}
}
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: run spotless on your PRs, so code is automatically formatted.

This is a previous miss from our side, we need to automate this.

17 changes: 17 additions & 0 deletions apps/opik-backend/src/main/java/com/comet/opik/api/AutomationRuleEvaluatorType.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
package com.comet.opik.api;

import com.fasterxml.jackson.annotation.JsonValue;
import lombok.AccessLevel;
import lombok.Getter;
import lombok.RequiredArgsConstructor;

@Getter
@RequiredArgsConstructor(access = AccessLevel.PRIVATE)
public enum AutomationRuleEvaluatorType {

LLM_AS_JUDGE("llm-as-judge"),
PYTHON("python");

@JsonValue
private final String type;
}
15 changes: 15 additions & 0 deletions apps/opik-backend/src/main/java/com/comet/opik/api/AutomationRuleEvaluatorUpdate.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
package com.comet.opik.api;

import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import com.fasterxml.jackson.databind.PropertyNamingStrategies;
import com.fasterxml.jackson.databind.annotation.JsonNaming;
import jakarta.validation.constraints.NotBlank;
import lombok.Builder;

@Builder(toBuilder = true)
@JsonIgnoreProperties(ignoreUnknown = true)
@JsonNaming(PropertyNamingStrategies.SnakeCaseStrategy.class)
public record AutomationRuleEvaluatorUpdate(
@NotBlank String code,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same, let's change code to a JsonNode field.

float samplingRate) {
}
167 changes: 167 additions & 0 deletions .../src/main/java/com/comet/opik/api/resources/v1/priv/AutomationRuleEvaluatorsResource.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,167 @@
package com.comet.opik.api.resources.v1.priv;

import com.codahale.metrics.annotation.Timed;
import com.comet.opik.api.AutomationRuleEvaluator;
import com.comet.opik.api.AutomationRuleEvaluatorUpdate;
import com.comet.opik.api.Page;
import com.comet.opik.domain.AutomationRuleService;
import com.comet.opik.infrastructure.auth.RequestContext;
import com.comet.opik.infrastructure.ratelimit.RateLimited;
import com.fasterxml.jackson.annotation.JsonView;
import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.headers.Header;
import io.swagger.v3.oas.annotations.media.Content;
import io.swagger.v3.oas.annotations.media.Schema;
import io.swagger.v3.oas.annotations.parameters.RequestBody;
import io.swagger.v3.oas.annotations.responses.ApiResponse;
import io.swagger.v3.oas.annotations.tags.Tag;
import jakarta.inject.Inject;
import jakarta.inject.Provider;
import jakarta.validation.Valid;
import jakarta.validation.constraints.Min;
import jakarta.validation.constraints.NotNull;
import jakarta.ws.rs.Consumes;
import jakarta.ws.rs.DELETE;
import jakarta.ws.rs.DefaultValue;
import jakarta.ws.rs.GET;
import jakarta.ws.rs.POST;
import jakarta.ws.rs.PUT;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.PathParam;
import jakarta.ws.rs.Produces;
import jakarta.ws.rs.QueryParam;
import jakarta.ws.rs.core.Context;
import jakarta.ws.rs.core.MediaType;
import jakarta.ws.rs.core.Response;
import jakarta.ws.rs.core.UriInfo;
import lombok.NonNull;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;

import java.net.URI;
import java.util.UUID;

@Path("/v1/private/automation/evaluator")
ldaugusto marked this conversation as resolved.
Show resolved Hide resolved
@Produces(MediaType.APPLICATION_JSON)
@Consumes(MediaType.APPLICATION_JSON)
@Timed
@Slf4j
@RequiredArgsConstructor(onConstructor_ = @Inject)
@Tag(name = "Automation rule evaluators", description = "Automation rule evaluators resource")
public class AutomationRuleEvaluatorsResource {

private final @NonNull AutomationRuleService service;
private final @NonNull Provider<RequestContext> requestContext;

@GET
@Path("/projectId/{projectId}")
Copy link
Collaborator

@andrescrz andrescrz Dec 20, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The path for the projectId should be project. In other words: /project/{projectId}

@Operation(operationId = "findEvaluators", summary = "Find Evaluators", description = "Find Evaluators", responses = {
@ApiResponse(responseCode = "200", description = "Evaluators resource", content = @Content(schema = @Schema(implementation = AutomationRuleEvaluator.AutomationRuleEvaluatorPage.class)))
})
@JsonView(AutomationRuleEvaluator.View.Public.class)
public Response find(@PathParam("projectId") UUID projectId,
@QueryParam("page") @Min(1) @DefaultValue("1") int page,
@QueryParam("size") @Min(1) @DefaultValue("10") int size) {

String workspaceId = requestContext.get().getWorkspaceId();
log.info("Looking for automated evaluators for project id '{}' on workspaceId '{}' (page {})", projectId,
workspaceId, page);
Page<AutomationRuleEvaluator> definitionPage = service.find(page, size, projectId);
log.info("Found {} automated evaluators for project id '{}' on workspaceId '{}' (page {}, total {})",
definitionPage.size(), projectId, workspaceId, page, definitionPage.total());

return Response.ok()
.entity(definitionPage)
.build();
}

@GET
@Path("/projectId/{projectId}/evaluatorId/{evaluatorId}")
ldaugusto marked this conversation as resolved.
Show resolved Hide resolved
@Operation(operationId = "getAutomationRulesByProjectId", summary = "Get automation rule evaluator by id", description = "Get dataset by id", responses = {
ldaugusto marked this conversation as resolved.
Show resolved Hide resolved
@ApiResponse(responseCode = "200", description = "Automation Rule resource", content = @Content(schema = @Schema(implementation = AutomationRuleEvaluator.class)))
})
@JsonView(AutomationRuleEvaluator.View.Public.class)
public Response getEvaluator(@PathParam("projectId") UUID projectId, @PathParam("evaluatorId") UUID evaluatorId) {
String workspaceId = requestContext.get().getWorkspaceId();

log.info("Finding automated evaluators by id '{}' on project_id '{}'", projectId, workspaceId);
AutomationRuleEvaluator evaluator = service.findById(evaluatorId, projectId, workspaceId);
log.info("Found automated evaluators by id '{}' on project_id '{}'", projectId, workspaceId);
ldaugusto marked this conversation as resolved.
Show resolved Hide resolved

return Response.ok().entity(evaluator).build();
}

@POST
@Operation(operationId = "createAutomationRuleEvaluator", summary = "Create automation rule evaluator", description = "Create automation rule evaluator", responses = {
@ApiResponse(responseCode = "201", description = "Created", headers = {
@Header(name = "Location", required = true, example = "${basePath}/api/v1/private/automation/evaluator/projectId/{projectId}/evaluatorId/{evaluatorId}", schema = @Schema(implementation = String.class))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please don't forget to update the 201 path example later.

})
})
@RateLimited
public Response createEvaluator(
@RequestBody(content = @Content(schema = @Schema(implementation = AutomationRuleEvaluator.class))) @JsonView(AutomationRuleEvaluator.View.Write.class) @NotNull @Valid AutomationRuleEvaluator evaluator,
@Context UriInfo uriInfo) {

String workspaceId = requestContext.get().getWorkspaceId();

log.info("Creating {} evaluator for project_id '{}' on workspace_id '{}'", evaluator.evaluatorType(),
evaluator.projectId(), workspaceId);
AutomationRuleEvaluator savedEvaluator = service.save(evaluator, workspaceId);
log.info("Created {} evaluator '{}' for project_id '{}' on workspace_id '{}'", evaluator.evaluatorType(),
savedEvaluator.id(), evaluator.projectId(), workspaceId);

URI uri = uriInfo.getAbsolutePathBuilder()
.path("/projectId/%s/evaluatorId/%s".formatted(savedEvaluator.projectId().toString(),
ldaugusto marked this conversation as resolved.
Show resolved Hide resolved
savedEvaluator.id().toString()))
.build();
return Response.created(uri).build();
}

@PUT
ldaugusto marked this conversation as resolved.
Show resolved Hide resolved
@Path("/projectId/{projectId}/evaluatorId/{id}")
ldaugusto marked this conversation as resolved.
Show resolved Hide resolved
@Operation(operationId = "updateAutomationRuleEvaluator", summary = "update Automation Rule Evaluator by id", description = "update Automation Rule Evaluator by id", responses = {
@ApiResponse(responseCode = "204", description = "No content"),
})
@RateLimited
public Response updateEvaluator(@PathParam("id") UUID id,
@PathParam("projectId") UUID projectId,
@RequestBody(content = @Content(schema = @Schema(implementation = AutomationRuleEvaluatorUpdate.class))) @NotNull @Valid AutomationRuleEvaluatorUpdate evaluatorUpdate) {

String workspaceId = requestContext.get().getWorkspaceId();
log.info("Updating automation rule evaluator by id '{}' and project_id '{}' on workspace_id '{}'", id,
projectId, workspaceId);
service.update(id, projectId, workspaceId, evaluatorUpdate);
log.info("Updated automation rule evaluator by id '{}' and project_id '{}' on workspace_id '{}'", id, projectId,
workspaceId);

return Response.noContent().build();
}

@DELETE
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're generally creating batch endpoints, so you can delete by sending multiple ids.

@Path("/projectId/{projectId}/evaluatorId/{id}")
ldaugusto marked this conversation as resolved.
Show resolved Hide resolved
@Operation(operationId = "deleteAutomationRuleEvaluatorById", summary = "Delete Automation Rule Evaluator by id", description = "Delete Automation Rule Evaluator by id", responses = {
@ApiResponse(responseCode = "204", description = "No content"),
})
public Response deleteEvaluator(@PathParam("id") UUID id, @PathParam("projectId") UUID projectId) {

String workspaceId = requestContext.get().getWorkspaceId();
log.info("Deleting dataset by id '{}' on workspace_id '{}'", id, workspaceId);
service.delete(id, projectId, workspaceId);
log.info("Deleted dataset by id '{}' on workspace_id '{}'", id, workspaceId);
return Response.noContent().build();
}

@DELETE
@Path("/projectId/{projectId}")
ldaugusto marked this conversation as resolved.
Show resolved Hide resolved
@Operation(operationId = "deleteAutomationRuleEvaluatorByProject", summary = "Delete Automation Rule Evaluator by Project id", description = "Delete Automation Rule Evaluator by Project id", responses = {
@ApiResponse(responseCode = "204", description = "No content"),
})
public Response deleteProjectEvaluators(@PathParam("projectId") UUID projectId) {

String workspaceId = requestContext.get().getWorkspaceId();
log.info("Deleting evaluators from project_id '{}' on workspace_id '{}'", projectId, workspaceId);
service.deleteByProject(projectId, workspaceId);
log.info("Deleted evaluators from project_id '{}' on workspace_id '{}'", projectId, workspaceId);
return Response.noContent().build();
}
}
79 changes: 79 additions & 0 deletions apps/opik-backend/src/main/java/com/comet/opik/domain/AutomationRuleDAO.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
package com.comet.opik.domain;

import com.comet.opik.api.AutomationRuleEvaluator;
import com.comet.opik.api.AutomationRuleEvaluatorUpdate;
import com.comet.opik.infrastructure.db.UUIDArgumentFactory;
import org.jdbi.v3.sqlobject.config.RegisterArgumentFactory;
import org.jdbi.v3.sqlobject.config.RegisterConstructorMapper;
import org.jdbi.v3.sqlobject.customizer.AllowUnusedBindings;
import org.jdbi.v3.sqlobject.customizer.Bind;
import org.jdbi.v3.sqlobject.customizer.BindList;
import org.jdbi.v3.sqlobject.customizer.BindMethods;
import org.jdbi.v3.sqlobject.statement.SqlQuery;
import org.jdbi.v3.sqlobject.statement.SqlUpdate;
import org.jdbi.v3.stringtemplate4.UseStringTemplateEngine;

import java.util.List;
import java.util.Optional;
import java.util.Set;
import java.util.UUID;

@RegisterArgumentFactory(UUIDArgumentFactory.class)
@RegisterConstructorMapper(AutomationRuleEvaluator.class)
public interface AutomationRuleDAO {

@SqlUpdate("INSERT INTO automation_rule_evaluators(id, project_id, workspace_id, evaluator_type, sampling_rate, code, created_by, last_updated_by) "+
"VALUES (:rule.id, :rule.projectId, :workspaceId, :rule.evaluatorType, :rule.samplingRate, :rule.code, :rule.createdBy, :rule.lastUpdatedBy)")
void save(@BindMethods("rule") AutomationRuleEvaluator rule, @Bind("workspaceId") String workspaceId);

@SqlUpdate("""
UPDATE automation_rule_evaluators
SET
sampling_rate = :rule.samplingRate,
code = :rule.code,
last_updated_by = :lastUpdatedBy
WHERE id = :id AND project_id = :projectId AND workspace_id = :workspaceId
""")
int update(@Bind("id") UUID id,
@Bind("projectId") UUID projectId,
@Bind("workspaceId") String workspaceId,
@BindMethods("rule") AutomationRuleEvaluatorUpdate ruleUpdate,
@Bind("lastUpdatedBy") String lastUpdatedBy);

@SqlQuery("SELECT * FROM automation_rule_evaluators WHERE id = :id AND project_id = :projectId AND workspace_id = :workspaceId")
Optional<AutomationRuleEvaluator> findById(@Bind("id") UUID id, @Bind("projectId") UUID projectId,
@Bind("workspaceId") String workspaceId);

@SqlQuery("SELECT * FROM automation_rule_evaluators WHERE id IN (<ids>) AND project_id = :projectId AND workspace_id = :workspaceId")
List<AutomationRuleEvaluator> findByIds(@BindList("ids") Set<UUID> ids, @Bind("projectId") UUID projectId,
@Bind("workspaceId") String workspaceId);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just merge this into a single method and overload the calls by wrapping the id into a set.


@SqlQuery("SELECT * FROM automation_rule_evaluators WHERE project_id = :projectId AND workspace_id = :workspaceId")
List<AutomationRuleEvaluator> findByProjectId(@Bind("projectId") UUID projectId, @Bind("workspaceId") String workspaceId);

@SqlUpdate("DELETE FROM automation_rule_evaluators WHERE id = :id AND project_id = :projectId AND workspace_id = :workspaceId")
void delete(@Bind("id") UUID id, @Bind("projectId") UUID projectId, @Bind("workspaceId") String workspaceId);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same about the deletion.


@SqlUpdate("DELETE FROM automation_rule_evaluators WHERE project_id = :projectId AND workspace_id = :workspaceId")
void deleteByProject(@Bind("projectId") UUID projectId, @Bind("workspaceId") String workspaceId);

@SqlUpdate("DELETE FROM automation_rule_evaluators WHERE id IN (<ids>) AND project_id = :projectId AND workspace_id = :workspaceId")
void delete(@BindList("ids") Set<UUID> ids, @Bind("projectId") UUID projectId, @Bind("workspaceId") String workspaceId);

@SqlQuery("SELECT * FROM automation_rule_evaluators " +
" WHERE project_id = :projectId AND workspace_id = :workspaceId " +
" LIMIT :limit OFFSET :offset ")
@UseStringTemplateEngine
@AllowUnusedBindings
List<AutomationRuleEvaluator> find(@Bind("limit") int limit,
@Bind("offset") int offset,
@Bind("projectId") UUID projectId,
@Bind("workspaceId") String workspaceId);

@SqlQuery("SELECT COUNT(*) FROM automation_rule_evaluators WHERE project_id = :projectId AND workspace_id = :workspaceId")
long findCount(@Bind("projectId") UUID projectId, @Bind("workspaceId") String workspaceId);

@SqlQuery("SELECT id FROM automation_rule_evaluators WHERE id IN (<ids>) and project_id = :projectId AND workspace_id = :workspaceId")
Set<UUID> exists(@BindList("ids") Set<UUID> ruleIds, @Bind("projectId") UUID projectId,
@Bind("workspaceId") String workspaceId);
ldaugusto marked this conversation as resolved.
Show resolved Hide resolved
}
Loading
Loading