Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge music branch to develop. #56

Merged
merged 82 commits into from
Nov 14, 2024
Merged

Merge music branch to develop. #56

merged 82 commits into from
Nov 14, 2024

Commits on Sep 1, 2023

  1. Configuration menu
    Copy the full SHA
    fc73bc9 View commit details
    Browse the repository at this point in the history

Commits on Sep 7, 2023

  1. Add translator dictionary, defining translation from internal shorten…

    …ed labels (model output) to more verbose format usable by `export_music.py`.
    vlachvojta committed Sep 7, 2023
    Configuration menu
    Copy the full SHA
    bda025a View commit details
    Browse the repository at this point in the history

Commits on Sep 13, 2023

  1. Configuration menu
    Copy the full SHA
    fe9aceb View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    2da0f24 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    2f661d2 View commit details
    Browse the repository at this point in the history

Commits on Sep 14, 2023

  1. Configuration menu
    Copy the full SHA
    5373283 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    1b9b676 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    7cc5293 View commit details
    Browse the repository at this point in the history
  4. Little refactoring.

    vlachvojta committed Sep 14, 2023
    Configuration menu
    Copy the full SHA
    fae66ce View commit details
    Browse the repository at this point in the history
  5. Add exporting music directly in parse_folder.py using config.ini

    …to define settings and `page_parser.py` to create music exporter object of `music/export_music/ExportMusicPage`.
    vlachvojta committed Sep 14, 2023
    Configuration menu
    Copy the full SHA
    b02ebb2 View commit details
    Browse the repository at this point in the history

Commits on Sep 18, 2023

  1. Configuration menu
    Copy the full SHA
    c07ba6a View commit details
    Browse the repository at this point in the history
  2. Add sorting music regions "in reading order" using y_min of boundin…

    …g box around polygon.
    vlachvojta committed Sep 18, 2023
    Configuration menu
    Copy the full SHA
    0afb958 View commit details
    Browse the repository at this point in the history

Commits on Oct 3, 2023

  1. Remove RegionCategory and LineCategory enums hard-coded in `layou…

    …t.py`. Get names only from Yolo `result.names`.
    vlachvojta committed Oct 3, 2023
    Configuration menu
    Copy the full SHA
    9371467 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    b733c74 View commit details
    Browse the repository at this point in the history

Commits on Oct 25, 2023

  1. Remove music exporter option from parse_folder.py and make `export_…

    …music.py` a stand-alone script.
    vlachvojta committed Oct 25, 2023
    Configuration menu
    Copy the full SHA
    b40e479 View commit details
    Browse the repository at this point in the history

Commits on Oct 27, 2023

  1. Add option to have more LineCroppers and ORC engines. Set every other…

    … text Layout engine to work only with 'text' lines.
    vlachvojta committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    7b93d16 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    50418dd View commit details
    Browse the repository at this point in the history

Commits on Nov 3, 2023

  1. Add PageLayout splitting enabling running multiple layout parsers eac…

    …h with its own setting and set of categories to work with.
    vlachvojta committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    8853046 View commit details
    Browse the repository at this point in the history
  2. Remove unused functions.

    vlachvojta committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    5ce86d1 View commit details
    Browse the repository at this point in the history
  3. Merge branch 'develop' into music

    # Conflicts:
    #	pero_ocr/core/layout.py
    vlachvojta committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    16e8ce3 View commit details
    Browse the repository at this point in the history

Commits on Dec 1, 2023

  1. Configuration menu
    Copy the full SHA
    3e2fcb8 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    a49409c View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    94bcae8 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    d688bc0 View commit details
    Browse the repository at this point in the history
  5. Refactor alto xml export.

    vlachvojta committed Dec 1, 2023
    Configuration menu
    Copy the full SHA
    65eacb5 View commit details
    Browse the repository at this point in the history

Commits on Dec 7, 2023

  1. Configuration menu
    Copy the full SHA
    bb88217 View commit details
    Browse the repository at this point in the history

Commits on Dec 14, 2023

  1. Minor updates

    ikiss-fit committed Dec 14, 2023
    Configuration menu
    Copy the full SHA
    3354a5b View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    59dd330 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    3c3322e View commit details
    Browse the repository at this point in the history
  4. New config section parsing and other changes after code review.

    - in layout_helpers.py add searching for a region by ID
    -  in page_parser.py change log error (missing crop) back to raise exception
    -  in page_parser.py change naming convention for config section names to {section_name}_\d+. All following examples are possible (with warnings) for OCR section: OCR, OCR_0, OCR_0_asdf...
    vlachvojta committed Dec 14, 2023
    Configuration menu
    Copy the full SHA
    27a82eb View commit details
    Browse the repository at this point in the history

Commits on Dec 22, 2023

  1. Add image_size to Yolo engine. Add config_get_list to get list of c…

    …ategories instead of json string.
    
    - `image_size` can be either int or (int, int) according to ultralytics docu: https://docs.ultralytics.com/modes/predict/#inference-arguments
    vlachvojta committed Dec 22, 2023
    Configuration menu
    Copy the full SHA
    ae516f4 View commit details
    Browse the repository at this point in the history
  2. Store box confidence (in LayoutExtractorYOLO) to RegionLayout and exp…

    …ort to page_xml region custom.
    vlachvojta committed Dec 22, 2023
    Configuration menu
    Copy the full SHA
    e5dd2b8 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    529234e View commit details
    Browse the repository at this point in the history
  4. Delete unwanted script.

    vlachvojta committed Dec 22, 2023
    Configuration menu
    Copy the full SHA
    036daf3 View commit details
    Browse the repository at this point in the history
  5. Add translating short music output to original encoding.

    - `TransformerEngineLineOCR` now translates all outputs according to `music_dictionary` dictionary in ocr.json (or omr.json), if exists.
    - Move `MusicTranslator` to a separate file for nicer imports everywhere.
    vlachvojta committed Dec 22, 2023
    Configuration menu
    Copy the full SHA
    697990c View commit details
    Browse the repository at this point in the history

Commits on Dec 29, 2023

  1. Configuration menu
    Copy the full SHA
    893baca View commit details
    Browse the repository at this point in the history
  2. Add CATEGORIES option to sorters and delete therefore unused functi…

    …on `sort_regions_in_reading_order`
    vlachvojta committed Dec 29, 2023
    Configuration menu
    Copy the full SHA
    e2a26de View commit details
    Browse the repository at this point in the history

Commits on Jan 4, 2024

  1. Configuration menu
    Copy the full SHA
    d59884a View commit details
    Browse the repository at this point in the history

Commits on Jan 15, 2024

  1. Tiny improvements.

    - print more informative warning message
    - add `__init__.py` to enable python package functionality for future.
    vlachvojta committed Jan 15, 2024
    Configuration menu
    Copy the full SHA
    0a1a0c0 View commit details
    Browse the repository at this point in the history

Commits on Jan 16, 2024

  1. Delete unused function from layout, music integration.

    - delete `layout.py/PageLayout.get_regions_of_category` as its functionality was substituted by `layout_helpers.py/split_page_layout_by_categories`. For sorting regions use (Naive|Smart)RegionSorter during `parse_folder` execution resulting in regions being sorted in xml output.
    - in `music_exporter.py` make translator optional argument.
    vlachvojta committed Jan 16, 2024
    Configuration menu
    Copy the full SHA
    faf0496 View commit details
    Browse the repository at this point in the history

Commits on Jan 18, 2024

  1. Configuration menu
    Copy the full SHA
    dd4bbc8 View commit details
    Browse the repository at this point in the history
  2. Change config categories, line_categories, add decoder filter.

    - `LINE_CATEGORIES`: for detected region also create a TextLine for future transcription.
    - `CATEGORIES`: categories to save in `LayoutExtractorYolo`, delete others. (if `None` or `[]`, save all)
    - disable YOLO detection results printing
    - filter categories for `PageDecoder`
    vlachvojta committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    6fc82e4 View commit details
    Browse the repository at this point in the history
  3. Rename MusicTranslator for to more general OutputTranslator and e…

    …verything around it.
    
    - substitution dictionary needs reversed format now. Translate key to values (to be more obvious). Example follows:
    - SSemantic music, Model output (now keys of dictionary): >2 + kGM + B3z + C4z + |..."
    - Semantic music (now values of dictionary): clef-G2 + keySignature-GM + note-B3_eighth + note-C4_eighth + barline...")
    vlachvojta committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    53ee27a View commit details
    Browse the repository at this point in the history

Commits on Jan 22, 2024

  1. Add option for rendering region categories for non-text regions.

    - enable by adding `--output-render-category` argument
    vlachvojta committed Jan 22, 2024
    Configuration menu
    Copy the full SHA
    68e7892 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    8ac485e View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    7032389 View commit details
    Browse the repository at this point in the history

Commits on Jan 23, 2024

  1. Configuration menu
    Copy the full SHA
    ef5f36f View commit details
    Browse the repository at this point in the history
  2. Add confidence estimation to PageOCR directly after detection. Update…

    … TextLine if new confidence is higher.
    
    - move `output_substitution` to PageOCR after confidence estimation.
    - add `TextLine.get_labels` for easier confidence estimation.
    - add `get_line_confidence_median` to save literally one line of code later...
    vlachvojta committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    8c738ee View commit details
    Browse the repository at this point in the history

Commits on Jan 26, 2024

  1. Add PageOCR.get_line_confidence solving problem of wrong confidence…

    … estimation.
    
    Problem was that confidence was estimated using `confidence_estimation.py/get_line_confidence` without cutting `log_probs` according to `line.logit_coords`. New method solves it.
    vlachvojta committed Jan 26, 2024
    Configuration menu
    Copy the full SHA
    949829a View commit details
    Browse the repository at this point in the history
  2. Unify logging style.

    vlachvojta committed Jan 26, 2024
    Configuration menu
    Copy the full SHA
    0b6e933 View commit details
    Browse the repository at this point in the history
  3. Update readme, remove translator.Semantic_to_SSemantic.json because i…

    …t was moved to ocr/omr.json (OCR engine config jon).
    vlachvojta committed Jan 26, 2024
    Configuration menu
    Copy the full SHA
    092b04f View commit details
    Browse the repository at this point in the history
  4. Improve translation of symbols in output_translator.py. Return orig…

    …inal symbol if not found in dictionary.
    vlachvojta committed Jan 26, 2024
    Configuration menu
    Copy the full SHA
    cc47eef View commit details
    Browse the repository at this point in the history

Commits on Jan 31, 2024

  1. Add atomic option to OutputTranslator + output substitution toggl…

    …es to `PageOCR` config section.
    
    - `SUBSTITUTE_OUTPUT` ('yes' or 'no'): enables (or disables) output substitution (enabled by default)
    - `SUBSTITUTE_OUTPUT_ATOMIC` ('yes' or 'no'): if 'yes' and any symbol cannot be translated, return original line. if 'no', line may be partially translated.
    vlachvojta committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    c7d90a1 View commit details
    Browse the repository at this point in the history

Commits on May 29, 2024

  1. Configuration menu
    Copy the full SHA
    d9430bc View commit details
    Browse the repository at this point in the history

Commits on May 30, 2024

  1. Configuration menu
    Copy the full SHA
    2f84712 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    eddf0e3 View commit details
    Browse the repository at this point in the history
  3. Add config parameter UPDATE_TRANSCRIPTION_BY_CONFIDENCE

    Parameter sets if PageOCR should update to new line:
     - every time (false)
     - only if better confidence (true)
    
    Applies in case of rerunning OCR on previously transcribed line)
    vlachvojta committed May 30, 2024
    Configuration menu
    Copy the full SHA
    70a7e35 View commit details
    Browse the repository at this point in the history

Commits on Jun 17, 2024

  1. Add ALTO baseline (export + import) in two options (float or points)

    - Versions older than 4.2 defines baseline as a simple float. (that's where the original baseline comes from)
    - version 4.2 and never defines baseline as a PointsType string with recommend format: "x1,y1 x2,y2 ..."
    vlachvojta committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    9dcd33f View commit details
    Browse the repository at this point in the history

Commits on Jun 19, 2024

  1. Add ALTO versions (options how to export baseline) + both baseline im…

    …port options.
    
    - Versions older than 4.2 defines baseline as a simple float. (baseline is exported as mean of all Y baseline points)
    - version 4.2 and never defines baseline as a PointsType string with recommend format: "x1,y1 x2,y2 ..."
    vlachvojta committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    1a46c00 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    2141002 View commit details
    Browse the repository at this point in the history
  3. Remove prints.

    vlachvojta committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    34c6584 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    82b3e70 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    2aed4bc View commit details
    Browse the repository at this point in the history

Commits on Jun 20, 2024

  1. Configuration menu
    Copy the full SHA
    4efdbab View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    134f51a View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    25f555c View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    26b0c1a View commit details
    Browse the repository at this point in the history

Commits on Jun 25, 2024

  1. Configuration menu
    Copy the full SHA
    3ce8bbc View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    3ec7902 View commit details
    Browse the repository at this point in the history

Commits on Jun 27, 2024

  1. Configuration menu
    Copy the full SHA
    520e3ae View commit details
    Browse the repository at this point in the history
  2. Set category filter fallback to [] for backward compatibility.

    Old XMLs on input don't have category => line.category = None, OCR (and others) have to be set to `[]` by default to process ALL PAGES.
    vlachvojta committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    9b414a0 View commit details
    Browse the repository at this point in the history
  3. Library versions fixes.

    1) Remove `ultralytics` and `music21` from dependencies for the whole projest. the user will have to install them when needed.
    2) Import `ultralytics` only when needed, so it doesn't create import error for specific numpy versions.
    
    Ultralytics has this dependency right now: "numpy>=1.23.5,<2.0.0". See current at [github.com/ultralytics/ultralytics/blob/main/pyproject.toml](https://github.com/ultralytics/ultralytics/blob/69cfc8aa228dbf1267975f82fcae9a24665f23b9/pyproject.toml#L67)
    vlachvojta committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    a110f0e View commit details
    Browse the repository at this point in the history

Commits on Jul 11, 2024

  1. Configuration menu
    Copy the full SHA
    f5a7a51 View commit details
    Browse the repository at this point in the history

Commits on Aug 5, 2024

  1. Fix bugs according to Pull request comment.

    In `smart_sorter.py`:
    - if less then to engines filtered, return original page_layout and not only the split one.
    
    In `music structures.py`:
    - change type of `lengths` to numpy array, fix min_length to take from numbers and not names.
    - ensure `encoded_group` is not None before appending it to the voice.
    
    full comment: [pero-ocr/pull/56/#issuecomment-2245202776](#56)
    vlachvojta committed Aug 5, 2024
    Configuration menu
    Copy the full SHA
    cc9b0ae View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    62af812 View commit details
    Browse the repository at this point in the history
  3. Add regions to splitting by category. If region.category set, move …

    …whole region to positive or negative (ignore categories of lines inside the region)
    vlachvojta committed Aug 5, 2024
    Configuration menu
    Copy the full SHA
    b60196f View commit details
    Browse the repository at this point in the history

Commits on Aug 6, 2024

  1. Add better None check.

    vlachvojta committed Aug 6, 2024
    Configuration menu
    Copy the full SHA
    4d2ddaa View commit details
    Browse the repository at this point in the history

Commits on Aug 27, 2024

  1. Disable exporting midi lines if no notes on the line.

    Export multirest as a simple default 'whole' rest.
    vlachvojta committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    7c4251e View commit details
    Browse the repository at this point in the history

Commits on Sep 24, 2024

  1. Configuration menu
    Copy the full SHA
    d9c64cd View commit details
    Browse the repository at this point in the history

Commits on Oct 16, 2024

  1. Simplify splitting page layouts to allow backwards (only look at regi…

    …on category, None = 'text')
    vlachvojta committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    fa1a897 View commit details
    Browse the repository at this point in the history

Commits on Oct 25, 2024

  1. Add IndexError to catch expression when calculating transcription con…

    …fidence -- in case when there are no logits (i.e. logits.shape[0] == 0) the confidence cannot be calculated.
    ikiss-fit committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    f5f2f42 View commit details
    Browse the repository at this point in the history

Commits on Nov 5, 2024

  1. Update layout.py

    michal-hradis authored Nov 5, 2024
    Configuration menu
    Copy the full SHA
    747e491 View commit details
    Browse the repository at this point in the history