Code

add document converters#332

Code

Merged

ArneBinder merged 31 commits intomainArneBinder/pytorch-ie:mainfrom document_convertersArneBinder/pytorch-ie:document_convertersCopy head branch name to clipboardSep 12, 2023

+859-76

Commits

Commits on Sep 11, 2023

• add document_converters

  ArneBinder

  committed
  bb110e4

• fix test and improve error message

  ArneBinder

  committed
  4e6b884

• fix naming

  ArneBinder

  committed
  de8ee7f

• fix documents.py

  ArneBinder

  committed
  23d1989

• improve convert_to()

  ArneBinder

  committed
  40d7a8a

• remove the document converters because they are not valid anymore

  ArneBinder

  committed
  a5d8a5e

• fix reset of document_converters

  ArneBinder

  committed
  0c1be09

• add target_document_type and document_converter to PieDatasetBuilder

  ArneBinder

  committed
  670002a

• fix register_document_converter()

  ArneBinder

  committed
  87fa88e

• allow a list for document_type and try to infer it from the converter if that is provided

  ArneBinder

  committed
  ec52457

• fix _infer_document_type_from_function_return() for strict=False

  ArneBinder

  committed
  8086438

• also allow derived document types when looking for registered converter

  ArneBinder

  committed
  8eadeee

• derive documents from each other

  ArneBinder

  committed
  dde2f9f

• rearrange document types

  ArneBinder

  committed
  698e6a0

• add DOCUMENT_TYPE and document_type property to TaskModule

  ArneBinder

  committed
  92d7db1

• just allow a single entry in document_type parameters

  ArneBinder

  committed
  d955c72

• PieDatasetBuilder accepts a document_converters parameter (instead of document_type and converter); rename convert_to() to to_document_type() and it accepts only a (mandatory) document_type parameter

  ArneBinder

  committed
  6999fa8

• fix IterableDataset.to_document_type()

  ArneBinder

  committed
  397c01e

• simplify dataset_to_document_type() and get_best_dataset_converter_with_types() to not accept a converter

  ArneBinder

  committed
  8861746

• infer document_type in (Iterable)Dataset.register_document_converter() instead of in DocumentDict version

  ArneBinder

  committed
  e7f5066

• fix DatasetDict.to_document_type()

  ArneBinder

  committed
  989c80c

• add log message

  ArneBinder

  committed
  256c7a0

• try to resolve the converter if it is a string

  ArneBinder

  committed
  0efc8b8

• add TextDocumentWithEntitiesAndRelations and TextDocumentWithEntitiesRelationsAndLabeledPartitions

  ArneBinder

  committed
  52548bb

• DatasetDict.to_document_type() does nothing if resolved_document_type == self.document_type

  ArneBinder

  committed
  0fd2048

Commits on Sep 12, 2023

• move test_get_pie_dataset_type() and remove import

  ArneBinder

  committed
  17607b3

• add tests for dataset builder

  ArneBinder

  committed
  99adce0

• add tests for (Iterable)Dataset.register_document_converter()

  ArneBinder

  committed
  54246da

• add tests for (Iterable)Dataset.to_document_type()

  ArneBinder

  committed
  bebc5d9

• add tests for DatasetDict.register_document_converter(); fix error message

  ArneBinder

  committed
  642bbd1

• add tests for DatasetDict.to_document_type()

  ArneBinder

  committed
  d4fc8f5