Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TopKMetrics not showable on evidently ui dashboard #1231

Open
CharlesCousyn opened this issue Jul 30, 2024 · 0 comments
Open

TopKMetrics not showable on evidently ui dashboard #1231

CharlesCousyn opened this issue Jul 30, 2024 · 0 comments

Comments

@CharlesCousyn
Copy link

CharlesCousyn commented Jul 30, 2024

In my project, I'm trying to set up a dashboard to display metrics for a search engine (ranking).

Here's a minimal example:

from evidently.ui.dashboards import DashboardPanelPlot, ReportFilter, PanelValue, PlotType, DashboardPanelCounter, CounterAgg
from evidently.renderers.html_widgets import WidgetSize
from evidently import ColumnMapping
from evidently import metrics
from evidently.ui.workspace import Workspace

from evidently.report import Report

#Create the workspace and project
ws = Workspace.create(path="./workspace_evidently")
project = ws.create_project("My test project")
project.save()

dataset = pd.DataFrame([
{
    "score": 0.5,
    "query": "my_query",
    "document_id": "docuemtn_id_1",
    "relevance": 0
},
{
    "score": 0.4,
    "query": "my_query",
    "document_id": "docuemtn_id_2",
    "relevance": 1
},
{
    "score": 0.3,
    "query": "my_query",
    "document_id": "docuemtn_id_5",
    "relevance": 0
},
{
    "score": 0.2,
    "query": "my_query",
    "document_id": "docuemtn_id_4",
    "relevance": 0
},
{
    "score": 0.1,
    "query": "my_query",
    "document_id": "docuemtn_id_3",
    "relevance": 1
}
])

report = Report(
    metrics=[
        metrics.MAPKMetric(4),
    ]
)

column_mapping = ColumnMapping(
    recommendations_type="score", target="relevance", prediction="score", item_id="document_id", user_id="query"
)
column_mapping.task = "recsys"
report.run(
    reference_data=None,
    current_data=dataset,
    column_mapping=column_mapping,
)
ws.add_report(project.id, report)

Let's add a dashboard like this:

project.dashboard.add_panel(
        DashboardPanelCounter(
            title="MAPKMetric",
            filter=ReportFilter(metadata_values={}, tag_values=[]),
            value=PanelValue(
                	metric_id="MAPKMetric",
                	field_path=metrics.MAPKMetric.fields.current,
                	legend="MAPKMetric",
                ),
            size=WidgetSize.HALF,
            agg=CounterAgg.SUM
        )
)
project.save()

If I now run evidently ui --workspace “./workspace_evidently” and go to the “Dashboard” page, I get the following error:

pydantic.v1.error_wrappers.ValidationError: 1 validation error for ParsingModel[float]
__root__
  value is not a valid float (type=type_error.float)

From what I understand, this is due to the fact that the TopKMetrics data are panda series showing the evolution of the metric with k ranging from 1 to 10 (useful for showing the curve). But it seems that, when building the Panel, there's no access to fetch the metric at the k I specified when creating my report metrics.MAPKMetric(4),. So, no float available when I do field_path=metrics.MAPKMetric.fields.... only a Series, which seems to broke the dashboard ui

mike0sv added a commit that referenced this issue Aug 26, 2024
emeli-dral pushed a commit that referenced this issue Aug 28, 2024
* fix recsys stuff
#1233
#1231

* fix tests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant