Submission removal

[alexpalms]

To keep track of the history of this discussion I am here reporting the discussion between our user Mscrnt (M) and myself alexpalms (A) on Discord:

(M): "Is it possible to have my submissions removed from the submission history in my dashboard page?
I was getting to know the program and submitted two of them thinking they were my own agent."

(A): "At the moment is not possible to remove a submission from the history, but you can (temporarily) filter them depending on their Status or use the search filter to show only the ones you are interested in.
But we want to hear our users and we do can add functionalities of this type to the platform.
Is there a specific reason why you would like to remove your submission?"

(M): "The goal would be just to have a clean dashboard. Otherwise, it isn't causing any harm. I don't really care about those submissions, so I'd rather they'd not be there."

(A): "Thanks a lot for sharing your opinion.
You are correct, they are not impacting your leaderboard positioning nor your achievements, as they will use the best scores achieved.
But all the submissions count for your statistics, meaning that you will accumulate points by keeping those submissions."

(M): "That last part seems like something I would want that feature for. If I was looking to keeps correct stats, that would be slightly annoying. Unless you can filter them out somehow."

(A): "Not from the stats, the stats are (at the moment) computed on all the successful submissions you have."

(M): "Ok, well that's good to know at least. Thanks for the info."

[alexpalms]

We discussed internally these aspects with the team, and we see two elements playing together here:

1. The desire to have a clean submission history in your own profile page to show publicly
2. The desire to have more control over what contributes to the stats calculation and what does not.

Let's try to address them in order.

Clean submission history
When we designed the submission history element, we thought about it as a log of all your submission activity, for this reason we did not provide methods to erase failed or "bad/wrong" submissions (whatever this means in different scenarios). You will see that it will happen often to have failed submissions, when you are submitting your own agent for the first time, dealing with your personal token, or your own dependencies docker image. So it is definitely normal. And having a log of them can also be useful to keep track of the errors and debugging them, in fact in every submission page there are error logs.

In addition with the search bar in that section and the filtering option on the status, you can easily navigate this log.

But we get that for a publicly shown element one may want to have something cleaner.

Here are some options that can be discussed/evaluated:

1. Allow the user to choose if he wants to make the whole submission history visible or hidden
2. Allow the user to explictly select which submissions to show publicly in his profile

Both solutions would still maintain the whole history of submissions as a persistent log and thus it would be visible for the user (at least).

Stats impact
In this regard, it is useful to clarify one aspect: at the moment there are 16 stats, in three categories:
A) cumulative (e.g. the sum of all submissions scores) - 7 stats
B) best (e.g. the longest round win streaks) - 9 stats
C) ratio (e.g. the number of perfects divided by the total number of evaluated episodes) - 3 stats

Of these three types, only the ratio type (so only 3 stats) is affected by "Bad/Wrong" submissions (but not by the failed ones that are not counted as valid), meaning that if you send a poorly performing agent (e.g. the random agent), it can diluite your stats, causing that number to go down.

But all the other 13 stats (type A and B) are either not affected (type B, best) or improved (type A, cumulative), so in the end a "wrong" submission is not harming your numbers.

Here are some options that can be discussed/evaluated (that can also be combined):

1. Completely remove the ratio stats from the platform, we are already thinking about this as we are not sure they provide a good metric for users. In this way, wrong submissions would cause no harm
2. Automatically filter out our pre-built example agents from the stats calculation

This would mean that all your valid submissions would then be considered for stats calculation but, as said, this would not harm type A and B stats.

Hoping this clarifies a bit the all situation, we would be very glad to hear your thoughts about these options.

Share your feedback!

PS: @discordianfish feel free to complete this discussion in case I missed something.

[mscrnt]

Hi Alex. Thanks for the follow up on this. I think the options you presented would work fine. The ability to allow the user to explicitly select which submissions to show publicly in his profile and automatically filtering out our pre-built example agents from the stats calculation might do the trick.

Here's how I image I'd like to use the dashboard. I see it as another social media dashboard. I get to show off the work I've done while not promoting the work I am not entirely proud of. Given the nature of the project we are working on, it would be nice to have robust and reliable stats. I don't want to have to do mental math to cut out pre-built or suboptimal submissions. From the user's pov, being able to delete unwanted submissions would be best. However, I understand the benefit of keep all submissions from your pov. Maybe it is possible to delete it from the dashboard, but keep it in a database in the backend?

Another nice feature to have would be the ability to comment on peoples submissions. Cheer them on or ask for advice or something. That's not part of this discussion though.

[alexpalms]

@mscrnt Thanks a lot for your very informative answer. We will iterate again on this and get back to you, but I wanted to ask if you could open a new feature request discussion regarding the last point you mentioned on commenting peoples submissions. You can use the space here: https://github.com/orgs/diambra/discussions/categories/2_feature-requests

[discordianfish]

What about showing only the best submissions for each game (and maybe difficulty etc)?

[mscrnt]

That might be an option. I kinda like the idea of a filter system that shows different lists. No Prebuilts, top 5, difficulty, game, character, etc... It would be nice if the stats changed based on the filter, too.

[discordianfish]

Yeah that seems like a good idea

[alexpalms]

Hi @mscrnt, so after an internal evaluation, we identified the following mods that should improve the UX, here they are:

For context, when in the following text I will refer to submission page, it is this type of pages: https://diambra.ai/submission/519

Profile and Submissions Public Visibility Quality
We totally agree that the personal profile and your own submissions are things one cares about as they are elements representing, for this reason we will:

• Add the possibility to check what is publicly visible in your profile, using a dedicated "Public preview" button
• Automatically remove failed submissions from public view (leaving them available in the private view)
• Embed the video recording in the submission page
• Improve the submission page metatags so that it will look as nice as possible when shared on social media

Stats reworking

• Removed ratio stats (that can get worse for bad performing submissions)
• Removed almost all cumulative stats except for one that cannot be improved by submitting the same agent over and over
• Added 5 Best result type of stats.
• So the new list of stats (for now) would be:
  • Those already available:
    • Most rounds won in a single episode
    • Most perfects in a single episode
    • Best global leaderboard position
    • Current position in the global leaderboard
    • Longest rounds streak
    • Characters having completed the game
    • Fewest steps to complete a game
  • New ones:
    • Best submission score in a single episode
    • Longest perfects streak
    • Most stages cleared in a single episode
    • Games with at least one completed submission
    • Games with at least one completed submission for each difficulty level

These modifications would neutralize the effect of random agents or bad performing agents on stats, meaning that only the best performing are considered and can "move" your stats.

Filters (stats/submissions)
We will add the following filters that would work on the stats and the submissions at the same time:

• All (no filter, stats from all submissions)
• Game filter (stats for a given game)
• Last 5 submissions (in chronological order)
• Difficulty filter (stats for a given difficulty)

In addition, we will also add (applicable) stats directly in the submission page, under the "Submission Result" section, as this can be a better way to keep track of the stats for those interested in seeing them per-submission.

What do you think? Do you have any feedback?

[mscrnt]

Wow. That's quite an impressive list of changes. I like all the changes you are planning. I didn't even consider the possibility of setting elements of your profile to private, but that would be a huge quality of life improvement. After looking at the submission page again, there are a couple more suggestions. Can it be possible to allow me to sort the submissions by column like Achievements or Agent? Also, the twitch video stream kind of feels like a pop up. Can that be loaded minimized? Maybe just highlight the button if there is a live stream active.

[alexpalms]

@mscrnt Thanks for the quick feedback and for the additional suggestions. I can confirm that the floating live window will have exactly the behavior you are describing, meaning that when the channel is not live the window will be minimized, when the channel is live and you have the window minimized, the button will blink. This is in a pull request ready to be merged in production.

Regarding the submission sorting, I believe is a great idea, I will get back to the team and ask for the feasibility but I am confident this will be deployable pretty soon.

I will update this discussion when all these mods will be live online.

[mscrnt]

Sounds great. I'll move any other off topic feedback to another discussion to keep this one clean.

[alexpalms]

@mscrnt We have a PR under review implementing all these changes, you can take a look at how the new website would appear by using this dev-subdomain: https://statistics.dev.diambra.ai/

Would be great if you could take a look at the stats in your dashboard and in the submissions pages, to validate what we implemented, we would really appreciate your feedback before releasing it!

[mscrnt]

Hi @alexpalms,

I can probably do a more targeted review with a change list, but here's what I got after a quick review.

Feedback on recent changes:

The "Public preview" button appears functional, but the differences from the regular view are minimal. Mostly personal info is removed, and some tooltips are gone.

I still see stats, even though I only have pre-built agents uploaded. I thought these weren't going to be included.

The sorting feature for columns like Achievements and Agent is fantastic, making organization easier.

The added filters for stats and submissions provide greater flexibility in accessing relevant data. However, I wonder how much use the "Number" filter will get now that I can sort by date. Also, were you going to include a filter for pre-built agents too?

The live stream window behaves as expected, remembering the last state and loading minimized when there's no live stream active. Not sure if that was even targeted for this release.

Suggestions:

Consider streamlining to just one view and removing the "Public preview" option for a smoother user experience.
Relocate personal info (real name, email) to an account page or privacy settings section.

Your team is doing a great job and I appreciate all the extra work you are putting in for this. Let me know if you need anything else.

[alexpalms]

Thanks a lot for the quick feedback @mscrnt, this is definitely very useful! Let me try to answer your points one by one:

Public preview

The "Public preview" button appears functional, but the differences from the regular view are minimal. Mostly personal info is removed, and some tooltips are gone.

We tried to implement what is done as a standard nowadays. You are correct, the differences are minimal, but:

1. It is good to have a clear way to see what is publicly shown, maybe for now are just a few things
2. you do not see failed submissions since they are filtered out, while they are visibile in the dashboard private view
3. you only see the best badges selected, not all the available ones

So in general, seems to be a good standard functionality to have that gives you feedback on what other view, with a minimal impact in the UI (just a button up there).

One important question though: which tooltips are you referring to when you say that they are gone in the public view?

Pre-build agents

I still see stats, even though I only have pre-built agents uploaded. I thought these weren't going to be included.

Yeah this evolved as mentioned in my previous message:

Given the stats reworking implented (only best results, removed cumulative stats, etc.) [...] these modifications would neutralize the effect of random agents or bad performing agents on stats, meaning that only the best performing are considered and can "move" your stats.

So yeah the pre-built agents are associated with stats, but they do not "pollute" them, meaning that as soon as you have an agent that perform better than the pre-built ones (which are just random agents) they will not influence your stats anymore. How do you see this point? Do you still feel pre-built agents should be filtered out from stats? and from public submission view?

Filters - Numbers

I wonder how much use the "Number" filter will get now that I can sort by date.

I see your point here, for the submission history, so we can probably remove that from there. But it could be useful to keep it in the stats filtering, right? Cause that allows you to see the stats for only the last 5 or 10 submissions (for the single submission you have them in the submission page directly). What do you think?

Filters - Pre-built

Also, were you going to include a filter for pre-built agents too?

This can probably be a middle ground, like filtering them out at will using the filters, but levaving them by default (in both submission history and stats). What do you think?

Others

Thanks a lot also for the suggestions, we will discuss them and evaluate how to proceed!

Looking forward to hearing from you.

[mscrnt]

Thanks a lot for explaining the details.

Public preview:
I get it now; the "Public preview" button serves its purpose well, offering a clear view of publicly visible content with minimal differences from the regular view. I don't have any failed submissions, yet. So I didn't notice it was getting filtered out. Keeping the "Public preview" functionality sounds good to me.

The tool tip I was referring to was the badges tool tip. And yeah, it do see how it only shows the selected badge.

Pre-built agents:
I think I didn't fully understand that pre-built agents were just random agents. I see how that works now. They don't need to be filtered out of public view.

Filters - Numbers:
Oh, right. Those filters also affect the stats. That's a good point. Leave it in. It might be nice to have more/different options. Top 3, for example. Just a nice to have.

Filters - Pre-built:
The middle-ground approach for the pre-built agent filter sounds good. Users can choose to filter them out using the available options, while still having them included by default in both submission history and stats.

Let me know if you need help with anything else.

[alexpalms]

Thanks a lot @mscrnt !!

Everything is clear now. We will implement the last suggestions you shared (including the tooltips) and deploy the new branch in production.

Will update and close this discussion when done!

[alexpalms]

@mscrnt All the modifications have been completed, they are visible in the temporary subdomain shared above (https://statistics.dev.diambra.ai/)
There is only one element left, the integration of the submission recording video in the submission page, I will leave this open until we add that, but we will be probably merge these changes first.

Do not hesitate to leave additional feedbacks if you want! Thanks for the support.

[mscrnt]

Looks great! Excellent job. Now time for me to get my own agent on there. I should have some time this next week to train for a few days straight.

Thanks again for all the hard work you and your team put into making this happen so quickly.

[alexpalms]

Super! Reach out in case you need support!

[alexpalms]

All these modifications have been rolled out!