[Feature] Image Annotator

[Nikhil-Nandagopal]

Summary

Users need a widget to annotate images.
A widget that can be used for labelling objects inside images. This is used for ML products.

Features:

Ability to draw inside image using a polygon
Store coordinates of the box and send it an API request

[riodeuno]

@Nikhil-Nandagopal As with the text annotator, please share your idea of the experience.

[Nikhil-Nandagopal]

https://github.com/Kunduin/react-picture-annotation#readme
https://github.com/UniversalDataTool/react-image-annotate

[wmdev0808]

Time Estimation:

• Research: 1
• POC development: 2
• Feedback, Refactor, test, documentation: 2

[somangshu]

The package used here internally has problems with re-rendering because of how the package internally uses memoize, Based on a few open issue we were able to find out how we could update the image, but we see this happening for all other properties as well, this might not be scalable for the future needs.

Now we have two other option before taking this up from scratch

1. https://github.com/Secretmapper/react-image-annotation - this package is very customisable, Build in JS, Has a few issues, Used by many in production
2. https://github.com/Kunduin/react-picture-annotation - Not very customisable, Written in typescript, Some issues, Active contributor

Based on a further research we will be able to identify what package we can go forward with

[somangshu]

@wmdev0808 kindly look into the packages mentioned above and do a POC to figure out what fits us best.

[wmdev0808]

@somangshu ,
I've tried to integrate two packages into our system.
Unfortunately, I've been stuck with Drag & Drop issue.
As you know, These two packages require users to select the region for an annotation.
But when I try to drag for the region selection, it leads to the entire widget movement.
Hope to hear how we can fix this issue

[somangshu]

@marks0351 is there a way to disable this for at-least while we are developing?

[marks0351]

@wmdev0808 as discussed in out last call, we can take some inspiration from input widget, it disables dragging the widget when it is focused.
https://github.com/appsmithorg/appsmith/blob/release/app/client/src/widgets/InputWidget/widget/index.tsx#L636

Also this proposed solution limits the user to moving the widget only by the widget name.
cc: @somangshu @momcilo-appsmith

[marks0351]

@wmdev0808 you cud also check how table widget columns are dragged to change order without the widget getting dragged.

[wmdev0808]

@somangshu , I've talked about the issue with other guys, but we could not figure it out

[wmdev0808]

@somangshu , @marks0351 provided a solution, so I am not blocked
The main problem is due to using the props directly, for usage in the underlying package, we are required to use state

[areyabhishek]

I was trying the widget today and I have a couple of questions:

1. How does some access the data being submitted? There is an onAnnotationSubmit event but how does one submit the annotation data in the first place?
2. I can't see any specific meta data in the entity explorer about the data being submitted.
3. Every time I click, there's a text box that appears with a submit button. How does one avoid using it? Not every annotation might need a text along with it.
4. Also the submit button on it doesn't convey that the data has been successfully submitted. It doesn't react much to my click.

[UXFalcon]

Design Handoff Link for Image Annotator

https://www.figma.com/file/w9CaXHF5iYcrLtiEbAHcFu/Original-Theme-Widgets-Library?node-id=1620%3A9821
@somangshu @Nikhil-Nandagopa

l

[wmdev0808]

Hi @areyabhishek,
Here are my answers to your questions:

1. How does some access the data being submitted? There is an onAnnotationSubmit event but how does one submit the annotation data in the first place?

• We are exposing annotations meta property, its default value is one from defaultAnnotations
• onAnnotationSubmit event is triggered when user select region, then add a text, finally click on Submit button, annotations will be filled with that data.
• how does one submit the annotation data in the first place?
  . With regard to this question, I am not sure exactly what it means

2. I can't see any specific meta data in the entity explorer about the data being submitted.

• I can see annotations meta property in the entity explorer

3. Every time I click, there's a text box that appears with a submit button. How does one avoid using it? Not every annotation might need a text along with it.

• Good point, at the moment there is no any annotation related packages to support such a scenario(underlying package's default behavior). For possibility, I need research on it

4. Also the submit button on it doesn't convey that the data has been successfully submitted. It doesn't react much to my click.

• Good point, the only way to check such a event, is to see if the current region is highlighted (underlying package's default behavior)

[Nikhil-Nandagopal]

@wmdev0808 the default annotation object is too complicated. Can we simplify its structure from

[
  {
    "geometry": {
      "type": "OVAL",
      "x": 33.85793871866295,
      "y": 25.568181818181817,
      "width": 27.409470752089135,
      "height": 28.977272727272723
    },
    "data": {
      "text": "Cat",
      "id": 0.30868057842924745
    }
  }
]

to

[
  {
    "shape": "OVAL",
    "x": 33.85793871866295,
    "y": 25.568181818181817,
    "width": 27.409470752089135,
    "height": 28.977272727272723,
    "text": "Cat"
  }
]

data and geometry don't need to be separate and we can rename the type property to shape to make it more intuitive
The expected structure also says there needs to be a "Selection" object in the annotation which should not be necessary for the user to configure

There's no way to delete or resize/move an annotation. This is important because most annotation work is data cleanup.
The onAnnotationSubmit should change to onAnnotationsChanged and be triggered whenever an annotation is added/resized/moved/deleted

The group drag / resize keeps getting triggered when I use the annotate option

We also need an option to let the user annotate from a drop-down instead of free form text because annotations are used to tag objects and free text is prone to tagging errors

[wmdev0808]

Hi @Nikhil-Nandagopal ,
I am not pretty sure about the last two sentences.
Please provide me detailed descriptions of them, accompanying screenshot or video

[Nikhil-Nandagopal]

@wmdev0808 check out the video
https://user-images.githubusercontent.com/3897254/144382077-de22a63f-de34-4202-8550-c3778560945e.mov

[wmdev0808]

Hi @areyabhishek and @Nikhil-Nandagopal,
From you guys' request, I think:

• The default annotation object is too complicated. We should simplify its structure
  • Already done
• There's no way to delete or resize/move an annotation. This is important because most annotation work is data cleanup.
  • This requires even more efforts to dig into the underlying package and customize/add such functionalities
• The group drag / resize keeps getting triggered when I use the annotate option
  • It should be fixed; @marks0351 , @jsartisan , I need you guys help with this
• We also need an option to let the user annotate from a drop-down instead of free form text because annotations are used to tag objects and free text is prone to tagging errors
  • This completely involves introducing features from react-image-annotate into current underlying package, react-image-annotation-ts.
  • Some researches into two underlying packages and accordingly customizing are required; More effort is required
  • it was an issue with rendering, where the internal property was not able to update, Or rather the change in property did not show the update on the widget component

[dilippitchika]

Updated 22 Dec 6PM IST
Had a preliminary discussion with @somangshu and @wmdev0808 to understand the current implementation of the widget. Below are the top priority requirements for the image annotator widget. Somangshu and Paul to evaluate and get back on the effort estimate for this.

Requirement	User story	Comments
Image the user sees must be visible fully, without having to stretch the image or resize the container. The images are loaded on top of a filled background to help fit better. See image attached.	As a user, I want to be able to view the full image so that I understand what needs to be annotated in the image	Currently the image stretches, we should not stretch it. Image should be contained in the widget like how the css object fit : contain works. Will need design as we may need to add a background color for the widget on which image loads
User can readjust the handles of the selection and move it just after dragging or select the section later	As a user, I want to be able to correct my annotation after I drag to be precise so that I can provide high quality training data	Need to understand how the annotations array will be updated. May need design
User can highlight/select a section in the image which is already annotated and delete it	As a user, I want to be able to delete my annotation if it’s not as I desired	Need to understand how the annotations array will be updated. May need design
Developers can choose to provide the annotator with an option to select from a list of choices or type in text for annotation	As a developer, I want to be able to allow select widget or input widget in the annotation popup to help users annotate faster and be consistent	This is just a starting point, so we may need to be flexible to add more widgets (at least selection) after this. Will need design.

[wmdev0808]

@somangshu,
For implementation on the requirements from @dilippitchika, I've done some research on the underlying package and alternatives.
Here are some result:

1. react-image-annotation

• Pros: It provides hooks for customization so that we can implement features such as edit / delete an annotation.
• Cons: It does not support resize/move, so we need to implement this features from scratch, hooking into the underlying package

2. react-picture-annotation

• Pros: It supports resize/move feature by default and provide a hook to customize input mechanism(text input, delete, select)
• Cons: Every annotation cannot overlap to be created, meaning we need a way to distinguish if the current user action is for movement of the existing one or creating a new one.

We are required to make decision which package for extension we will be using as an underlying package.
After that, if someone can provide appropriate solutions, tips, suggestions for required functionalities, it will be very helpful for me.

[somangshu]

@wmdev0808 based on the your comment above, Here are my thoughts

• I remember us using react-image-annotation over react-picture-annotation for some reason, Probably typescript support, What is the caveat here? Are we talking a tech debt?
• The con listed in react-picture-annotation is a UX issue which might be true in any implementation, if that is not an issue, this package can be preferred
• @dilippitchika in case having options like edit/delete suffice the resize/move function, we could go for it under what is already implemented.
  • @wmdev0808 what do mean by edit and resize, what are we allowing to edit here, the input label?
• @wmdev0808 can we give the functionality to the appsmith dev to change input type in the react-image-annotation package?

Once these questions are answered, we can move ahead.
cc @sbalaji1192

[wmdev0808]

@somangshu,

• react-picture-annotation supports TS by default.

We just choose react-image-annotation for popularity(github star) and more functionalities(for example support for various selector types, RECTANGLE, OVAL, POINT, even custom one, however, react-picture-annotation just supports RECTANGLE)

• edit: label input, tag input on a newly created annotation or existing annotation

• resize: mouse interactions on boundaries of an annotation, resulting in geometric value changes

• I think that adding an option for input type(text input or select input) is possible

[somangshu]

@wmdev0808 thanks for the clarity on this

@dilippitchika we would like to have a discussion around this with you

cc @sbalaji1192

[dilippitchika]

@somangshu @wmdev0808 We can have a discussion on this. I also have some other details to share.

I was checking react-image-annotation and the resize issue is open since the last 2+ years. I don't there's any improvement on that front.

I still think it's common to have the resize and move feature for annotation projects.

Can compromise non rectangular shapes as Most ML training data deals with rectangular bounding boxes. Will block time for discussion.

[dilippitchika]

Discussions with @wmdev0808 and @somangshu today

1. Dilip to create an updated PRD and share by Monday. PRD to cover the select/text functionality along with the above discussed requirements.
2. Requirements are same as the ones discussed above
3. Not in scope : Overlapping annotations in an existing bounding box, non-rectangular bounding boxes.
4. Decided to move ahead with react-picture-annotation
5. We will close the current PR and work on a new one.
6. Current effort estimate suggested was 4days including tests

[dilippitchika]

Updated PRD

Objective

Image annotation is useful for generating training data for machine learning projects. Companies/Teams employ resources internally or work with external vendors to annotate values from documents or images.

Our image annotation widget can be used to annotate images by app viewers to generate training data.

Success Metrics *

Goal	Metric
Users can create ML annotation use-cases	No. of deployed apps using the widget

Requirements

Requirement	User story	Comments
Image the user sees must be visible fully, without having to stretch the image or resize the container. The images are loaded on top of a filled background to help fit better. See image attached.	As a user, I want to be able to view the full image so that I understand what needs to be annotated in the image	Currently the image stretches, we should not stretch it. Image should be contained in the widget like how the css object fit : contain works. Will need design as we may need to add a background color for the widget on which image loads
User can readjust the handles of the selection and move it just after dragging or select the section later	As a user, I want to be able to correct my annotation after I drag to be precise so that I can provide high quality training data	Needs design
User can highlight/select a section in the image which is already annotated and delete it	As a user, I want to be able to delete my annotation if it’s not as I desired	Needs design
Developers can choose to provide the annotator with an option to select from a list of choices or type in text for annotation	As a developer, I want to be able to allow select widget or input widget in the annotation popup to help users annotate faster and be consistent	Needs design

Out of Scope

• Overlapping annotations
• Non-rectangular bounding boxes
• Multi-select or tags for annotation
• More than one annotation fields for an image

Developer Handoff Document in Figma
Needs Design Team. This example is not final, just a mock to help dev team visualise the change, look at the updated pane in image annotation widget. https://www.figma.com/file/YtmxcUEmBraZNd1jhxBL2L/Original-Theme-Widgets-Library-(Copy)?node-id=428%3A3414

Property pane updated

[momcilo-appsmith]

@somangshu Why is there a "Needs Design" label here?

[somangshu]

Needs design was originally added to make this widget consistent with other. Now that we will be starting from the scratch in the future, it might not be required

[Nikhil-Nandagopal]

Lots of new information here about this widget
#19782

[Nikhil-Nandagopal]

We recently launched our custom widget which allows you to create an image annotator!
Check out the post below where you can fork the widget and start using it.
https://community.appsmith.com/template/image-labeler