Migrations always generated for custom type with DBAL 4 #1441

michnovka · 2024-07-03T11:48:16Z

Bug Report

Q	A
BC Break	no
Migrations version	3.8.0
Migrations bundle	3.3.1
ORM	3.2.1
DBAL	4.0.4
Symfony	7.1.2

Summary

I have issue with DBAL 4 and custom type, the migrations keep getting generated again and again. Furthermore, the down migration looks totally bogus. This is possibly related to #1435

I know that DBAL 4 dropped requiresSqlHint (in doctrine/dbal#5107 , afterwards some issues were found and fixed - doctrine/dbal#6257 )

So when I am using custom type, I expect the first migration diff to drop the DC2Type comments. However my tables have these fields already dropped and yet the migration is being generated.

enum ActionType: string
{
    case SUBMIT = 'submit';
    case CANCEL = 'cancel';
}

class ActionTypeType extends Type
{
    public function getSQLDeclaration(array $column, AbstractPlatform $platform): string
    {
        $class = ActionType::class;

        $values = [];

        foreach ($class::cases() as $val) {
            $values[] = "'{$val->value}'";
        }

        return "enum(" . implode(", ", $values) . ")";
    }

    public function convertToDatabaseValue($value, AbstractPlatform $platform): mixed
    {
        $class = ActionType::class;
        if ($value !== null && !($value instanceof BackedEnum)) {
            $value = $class::tryFrom($value);
        }else{
            return null;
        }

        return $value->value;
    }

    public function convertToPHPValue($value, AbstractPlatform $platform): ?BackedEnum
    {
        if ((!is_int($value)) && !is_string($value)) {
            return null;
        }

        return $class::tryFrom($value);
    }
}

I then have my entity as

#[ORM\Entity(repositoryClass: ActionRepository::class)]
#[ORM\Table(name: 'actions')]
class Action
{
    #[ORM\Id]
    private string $name;

    #[ORM\Column(type: ActionType::class, nullable: false)]
    private ActionType $type;
}

and in Kernel.php I set up type mapping inside Kernel::process():

$typesDefinition[ActionType::class] = ['class' => ActionTypeType::class];
$container->setParameter('doctrine.dbal.connection_factory.types', $typesDefinition);

Now I know that the types are assigned correctly, as migrations generate up like this:

    public function up(Schema $schema): void
    {
        $this->addSql('ALTER TABLE actions CHANGE type type enum(\'submit\', \'cancel\') NOT NULL');
    }

but it is generated ALWAYS.

and the down() migration looks even weirder:

    public function down(Schema $schema): void
    {
        $this->addSql('ALTER TABLE actions CHANGE type type VARCHAR(0) NOT NULL');
    }

Everything works fine with DBAL 3, which uses SQL comments

The text was updated successfully, but these errors were encountered:

michnovka · 2024-07-03T14:38:20Z

@greg0ire I am tagging you as you authored some of the relevant changes and I know you have deep insight into removing comments in DBAL 4.

I was digging into the code, the issue lies here:

So Doctrine\Migrations\Generator\DiffGenerator::generate() calls createFromSchema to do comparison of old vs new schema. This calls Doctrine\DBAL\Schema\MySQLSchemaManager (which inherits AbstractSchemaManager) and specifically it fails on function _getPortableTableColumnDefinition(). Where the array $tableColumn which this function gets is

 array:11 [
  "table_name" => "actions"
  "field" => "type"
  "type" => "enum('submit','cancel')"
  "null" => "NO"
  "key" => ""
  "default" => null
  "extra" => ""
  "comment" => ""
  "characterset" => "utf8mb4"
  "collation" => "utf8mb4_general_ci"
  "name" => ""
]

This function then calls $type = $this->platform->getDoctrineTypeMapping($dbType) and since enum, or whatever custom type is not recognized, it defaults to StringType.

Now check DBAL 3 code:
https://github.com/doctrine/dbal/blob/893417fee2bc5a94a10a2010ae83cab927e21df3/src/Schema/MySQLSchemaManager.php#L189-L192

This uses the comment to deduce the proper type and override whatever type is in DB. Without this now, DBAL considers any unknown DB type to be StringType and I have no way how to tell it its not a string type. Hence the migration will always consider old schema column to be string.

Now the new schema uses the DiffGenerator::createToSchema() which does not use Doctrine\DBAL\Schema\AbstractSchemaManager but instead Doctrine\Migrations\Provider\OrmSchemaProvider and this uses under the hood EntityManager which of course identifies the column type properly as ActionTypeType or whatever custom type you use.

So, since we removed comments in DBAL 4, how can the fromSchema be generated properly? This uses only DB values, it does not care about type definition in Entity code (thats relevant for toSchema only).

greg0ire · 2024-07-03T14:47:03Z

Please upgrade the ORM to 3.8.5: doctrine/dbal#6423

EDIT: I shouldn't answer so hastily

michnovka · 2024-07-03T15:22:11Z

@greg0ire I am using DBAL 4.0.4 which is based on 3.8.6 that includes the linked fix. ORM is at 3.2.1 and there is no newer version. And the issue is ONLY with DBAL 4 not DBAL 3

berkut1 · 2024-07-03T19:06:38Z

@michnovka It seems that when Doctrine reads the schema of your database, the getSQLDeclaration function is not being called, which is why it cannot recognize the type. There might be a bug in https://github.com/doctrine/dbal/blob/4.0.x/src/Platforms/AbstractMySQLPlatform.php

Try debugging this function to determine at which stage the type is being lost:
https://github.com/doctrine/dbal/blob/90424473eb144659a89fb1b5c9bca37297085351/src/Platforms/AbstractPlatform.php#L2190
Probably, this is a mistaken assumption. Here, two schemas columns are being compared. If initially it was read as StringType, then the error is somewhere earlier.

michnovka · 2024-07-03T19:21:33Z

@berkut1 this is indeed a mistaken assumption imo, as getSQLDeclaration not being called is a consequence of not having the proper type. The type has to be known in order for that function to be called. And it is called, just for the new schema.

The issue is that oldSchema is created based on database ONLY. It ignores PHP code. And thats good, as the php code defines the new schema, if it took the PHP code as a hint of the type, then migration has no point, as it would be the same type always.

I honestly see no other way than to somehow store in DB the name of the custom type which was used. Which is exactly what the DB comment was used for. WDYT?

berkut1 · 2024-07-03T20:11:11Z

@michnovka
No, DC2Type comments have been useless since around DBAL 3.x. Doctrine can determine custom types on its own if they are properly registered. You need to find out why, in your case, this is not happening for Enum and where exactly the error is in the code.

michnovka · 2024-07-03T20:19:11Z

@berkut1 how are they useless when even in 3.9.x there is this code:

https://github.com/doctrine/dbal/blob/893417fee2bc5a94a10a2010ae83cab927e21df3/src/Schema/MySQLSchemaManager.php#L189-L192 ?

And this is where it fails, this is where in 3.8.x version it assigned properly custom type, but in 4.0.x it assigns StringType (as the code assigning type based on comment is missing)

And again, think about what I observed above - the oldSchema is created based on DB data ONLY. not PHP. Only DB. So if DB has no indication about the type, how can it be properly determined for nonstandard types?
newSchema uses PHP code to determine types. And thats good.

Doctrine can determine custom types on its own if they are properly registered.

This has no sense for migrations. If I had customType1 field in table, and I changed it to customType2 field, then did migration, how would doctrine deduce it was before customType1? It is no longer defined and registered. PHP code always reflects only the newSchema.

berkut1 · 2024-07-03T20:24:36Z

DBAL 3.x should have backward compatibility, that why it can works with DC2Types and without. there was added this https://github.com/doctrine/dbal/blob/893417fee2bc5a94a10a2010ae83cab927e21df3/src/Platforms/AbstractPlatform.php#L108
that people can smoothly migrate from DBAL3 to DBAL4

michnovka · 2024-07-03T20:30:36Z

With DBAL 3.8.X and doctrine.dbal.disable_type_comments: true the issue is also present.

And of course it is. Read again where and why it happens, I explained exactly where the issue lies. oldSchema is created based on DB details only. If we dont use comments (either as they are removed in DBAL 4, or if we disable them manually in DBAL 3) then oldSchema will have no idea what type was used before.

Thanks for your interest and will to help.

berkut1 · 2024-07-03T20:47:25Z

Here,
https://github.com/doctrine/dbal/blob/90424473eb144659a89fb1b5c9bca37297085351/src/Schema/MySQLSchemaManager.php#L135
and
https://github.com/doctrine/dbal/blob/90424473eb144659a89fb1b5c9bca37297085351/src/Schema/MySQLSchemaManager.php#L218
Doctrine is trying to map your registered custom types to your database. What happens when it tries to map the Enum type in your case?

UPD:
You also can check if your type registered, here:

michnovka · 2024-07-03T20:57:08Z

https://github.com/doctrine/dbal/blob/90424473eb144659a89fb1b5c9bca37297085351/src/Schema/MySQLSchemaManager.php#L135

dump($type, $dbType);

gives

1 "string"
2 "enum"

and

https://github.com/doctrine/dbal/blob/90424473eb144659a89fb1b5c9bca37297085351/src/Schema/MySQLSchemaManager.php#L218

dump($column);

gives

Doctrine\DBAL\Schema\Column {#926
  #_name: "type"
  #_namespace: null
  #_quoted: false
  #_type: Doctrine\DBAL\Types\StringType {#636}
  #_length: 0
  #_precision: null
  #_scale: 0
  #_unsigned: false
  #_fixed: false
  #_notnull: true
  #_default: null
  #_autoincrement: false
  #_platformOptions: []
  #_columnDefinition: null
  #_comment: ""
}

UPD:
I checked and the typeRegistry does contain my custom type ActionTypeType

greg0ire · 2024-07-03T21:03:42Z

@michnovka it's unclear to me whether you know what platform-aware comparison is. In case you don't, since DBAL 3.2.0, platform-aware comparison allows to compare the generated SQL instead of comparing schemas. This means that 2 different schemas could result in an empty diff, if they result in the same SQL being generated. More on this here: https://www.doctrine-project.org/2021/11/26/dbal-3.2.0.html

Sorry if you already know this, it's genuinely hard to tell for me.

michnovka · 2024-07-03T21:15:12Z

@greg0ire yes, I get this.

The problem is that when the comparator tries to compare the schemas, it is working with oldSchema and newSchema. oldSchema is Schema instance which is created and populated with data using Doctrine\DBAL\Schema\MySQLSchemaManager and it considers my column to be StringType (cos DB column definition is enum(...) which it doesnt understand, so it says, its StringType - this can be seen in my debug output in comment #1441 (comment)).

This means that when it tries to compare SQL generated between old and new schemas, it calls StringType::getSQLDeclaration output instead of the ActionTypeType::getSQLDeclaration for the old schema.

And this is expected. As if in DB there is no info about which custom type the column is, how can the MySQLSchemaManager assume its ActionTypeType and then gets its SQL from ActionTypeType::getSQLDeclaration to compare with new schema.

Now the new schema is another Schema instance, but this one is created using Doctrine\Migrations\Provider\OrmSchemaProvider and this calls internally EntityManager and that uses PHP code and properly considers the column ActionTypeType. Then new schema's SQL is the proper one, while the old ones is nonsense, as it thinks its a StringType.

greg0ire · 2024-07-03T21:27:09Z

Ah I get it now, thanks for explaining it again. I think this means the issue is that the DBAL does not understand enum(…), and cannot find a type that will produce the same DDL. I don't see a way around that. I now better get the point of doctrine/dbal#6444, which I now realize was about the same thing.

michnovka · 2024-07-03T21:32:16Z

yes, it all comes to a simple problem - the oldSchema is generated using ONLY DB introspection. And if DB contains no info about which type was used, then it cannot call proper getSQLDeclaration function.

And no, the problem does not simply limit to enum. I can make an example with another custom type which will have the same issue without enums. ALL custom types have this issue in fact, i.e. all custom types with custom getSQLDeclaration which differs from those provided by DBAL package.

berkut1 · 2024-07-03T21:35:12Z

Here, Doctrine is reading your database. Does it also read the type as a string here?
https://github.com/doctrine/dbal/blob/90424473eb144659a89fb1b5c9bca37297085351/src/Schema/AbstractSchemaManager.php#L195

leads to https://github.com/doctrine/dbal/blob/90424473eb144659a89fb1b5c9bca37297085351/src/Schema/MySQLSchemaManager.php#L344

greg0ire · 2024-07-03T21:39:14Z

This looks related: doctrine/dbal#5308 (comment)

michnovka · 2024-07-03T21:46:30Z

Here, Doctrine is reading your database. Does it also read the type as a string here? https://github.com/doctrine/dbal/blob/90424473eb144659a89fb1b5c9bca37297085351/src/Schema/AbstractSchemaManager.php#L195

No, here it fetches it from DB like this:

   1 => array:10 [
      "TABLE_NAME" => "actions"
      "field" => "type"
      "type" => "enum('submit','cancel')"
      "null" => "NO"
      "key" => ""
      "default" => null
      "EXTRA" => ""
      "comment" => ""
      "characterset" => "utf8mb4"
      "collation" => "utf8mb4_general_ci"
    ]

But this is just one step before _getPortableTableColumnDefinition is called which is where the $this->platform->getDoctrineTypeMapping($dbType); is called, and which we already addressed and debugged in #1441 (comment)

And @greg0ire this is where the issue lies:
https://github.com/doctrine/dbal/blob/90424473eb144659a89fb1b5c9bca37297085351/src/Schema/MySQLSchemaManager.php#L218

        $column = new Column($tableColumn['field'], Type::getType($type), $options);

specifically the Type::getType($type). Without comments, there is no way at all for the old type to be "guessed" correctly. If I use custom type which has SQL declaration always as int(3), (lets call it CustomInt3Type), then the Column instance will always be deduced as IntegerType and for comparison IntegerType::getSQLDeclaration will be called to be compared against newSchema's CustomInt3Type::getSQLDeclaration. There is just no way for it to know that this int(3) is actually CustomInt3Type and not IntegerType

berkut1 · 2024-07-03T22:00:32Z

Ah, I understand now. It calls Type::getType with enum and cannot find such a type because it doesn't exist.
In Symfony, you can work around this by registering another type to let Doctrine know about it: https://symfony.com/doc/current/doctrine/dbal.html#registering-custom-mapping-types-in-the-schematool (this instruction correct for DBAL3)
In your case, for DBAL 4 with Symfony, you need to do it like this:

        mapping_types:
            enum: ancestor_enum 
        types:
            ancestor_enum : 'App\Model\AncestorEnumType'

Unfortunately, I can't help with how to do this without Symfony.

UPD:
In the example, I specified the type as ancestor, assuming that you can register it in the mapping (mapping_types) only once. For all your other Enums, it will be enough to inherit from it and register only as a custom type (types). Otherwise, you would have to duplicate entries for all Enums in mapping_types.

michnovka · 2024-07-04T09:04:16Z

@greg0ire as a temporary workaround, how can I tell migration to ignore certain columns? Thanks!

greg0ire · 2024-07-04T09:17:01Z

Maybe you can use asset filtering, but I don't think it works at the column level.

michnovka · 2024-07-04T11:56:46Z

So the issue is this:

we get table columns in raw format from DB using DBAL\Schema\AbstractSchemaManager::fetchTableColumnsByTable() This is an array like

   1 => array:10 [
      "TABLE_NAME" => "actions"
      "field" => "type"
      "type" => "enum('submit','cancel')"
      "null" => "NO"
      "key" => ""
      "default" => null
      "EXTRA" => ""
      "comment" => ""
      "characterset" => "utf8mb4"
      "collation" => "utf8mb4_general_ci"
    ]

This is converted to DBAL\Schema\Column instance, which has incorrect type (as DBAL has no idea what custom type is to be used without comments, so it guesses best match from default types)
then Column::getType()::getSQLDeclaration() is called to obtain incorrect SQL

And on the new schema, we get correct type for column and call its getSQLDeclaration()

These do not match.

Proposed solution:

Why not leverage Column::$_columnDefinition for the old schema? Lets populate this field inside AbstractSchemaManager::_getPortableTableColumnDefinition.

I dont understand why AbstractPlatform::columnsEqual() unsets this. But if we dont unset it and specify it explicitly for old schema, the comparison would be done properly

@greg0ire WDYT?

greg0ire · 2024-07-04T13:50:24Z

Sounds good, feel free to give it a try, and see if anything breaks 👍

michnovka · 2024-07-05T14:56:13Z

I tried to go this way and I am afraid this would create tons of unexpected issues. There are specific test cases that ensure that

Schema diff is empty, since only columnDefinition changed from null (not detected) to a defined one

AbstractComparatorTestCase::testWillNotProduceSchemaDiffOnTableWithAddedCustomSchemaDefinition

so for whatever reason this is considered important. Even the comment in the AbstractPlatform::columsEqual talks about this specifically to ignore the column definition.

So maybe a solution would be to add comments back so that the type can be deduced properly?

greg0ire · 2024-07-05T20:25:12Z

I don't think that's the direction we want to go, no. Re-reading the cookbook you mentioned I see

In this case however Schema-Tool update will have a hard time not to request changes for this column on each call.

It seems like that might have been fixed in doctrine/dbal#5224, which might make solution 1 the way to go here.

berkut1 · 2024-07-06T00:09:24Z

@michnovka
I don't know why you ignored my solution #1441 (comment). For reference, I had a very similar problem, not with enum, but with the inet type, which is unknown to Doctrine. It tried, as in your case, to convert it to VARCHAR in the down() method and constantly generated diff migrations.

As I mentioned in my comment on the instructions, I solved the problem for myself in this way:

doctrine:
    dbal:
        mapping_types:
            inet: my_inet

        types:
            my_inet: { class: 'App\Model\InetType' }

I want to clarify, as stated in the instructions at https://www.doctrine-project.org/projects/doctrine-orm/en/3.2/cookbook/mysql-enums.html, it is recommended to do it like this

$conn->getDatabasePlatform()->registerDoctrineTypeMapping('enum', 'string');

but this solution works only for DBAL3. For DBAL4, you need to refer not to string but to your newly registered custom type, as I indicated in my previous comment.

However, if you haven't tried this solution at all, you can first refer to string; it might work with enum.

michnovka · 2024-07-07T11:15:00Z

@berkut1 Thanks for the suggestion, but this is not a solution for my case.

It works in your case when you have 1:1 mapping between DB type INET and your class MY_INET. So your code is 100% certain, when it finds INET type in DB to use that specific type.

But in my case, every ENUM is different. The example I provided is ENUM('submit','cancel') and it uses custom type ActionTypeType extends AbstractEnumType. AbstractEnumClass is my base class for enums, and I have many others. There is e.g. ColorEnumType extends AbstractEnumType which creates ENUM('black','red'). So DB ENUM type maps to multiple PHP Type classes.

berkut1 · 2024-07-07T17:05:28Z

@michnovka

Yes, I understand, which is why I initially suggested creating an abstract Enum class and registering it, then inheriting from it for custom types.
If I understand correctly how Doctrine works with types, when comparing schemas, it only compares the types of SQL, and the comparison of ColumnDeclarationSQL happens only at the final stage: https://github.com/doctrine/dbal/blob/90424473eb144659a89fb1b5c9bca37297085351/src/Platforms/AbstractPlatform.php#L2199-L2204

In other words, you need to make Doctrine understand that it has a base Enum class for comparison, with which it will compare all the descendants of this class.
This test for DBAL3 with disabled type comments, show the logic:
https://github.com/doctrine/dbal/blob/893417fee2bc5a94a10a2010ae83cab927e21df3/tests/Platforms/AbstractPlatformTestCase.php#L1448

michnovka · 2024-09-28T14:03:26Z

has anybody started work on the Verbatim type? I might have some time in end of October/November to look into this.

PowerKiKi · 2024-09-28T14:16:55Z

I thought I would give it a try, but I didn't yet. And I won't have much time for a while. Feel free to start whenever you want. And notice everybody here when you do, so that we don't duplicate work.

morozov · 2024-10-08T08:29:48Z

Linking a couple more existing issues for reference: doctrine/dbal#4470, doctrine/dbal#5306.

VincentLanglet · 2024-10-08T16:05:28Z

I encounter the same issue with my custom types.
I keep generating dozen of lines like

$this->addSql('ALTER TABLE activity_events CHANGE status status ENUM(\'DISPATCHED\', \'ONGOING\', \'SUCCEEDED\', \'FAILED\') NOT NULL');

in my migration diff, when it worked fine with COMMENT \'(DC2Type:enumActivityEventStatus)\''

It does not give a good experience with the DBAL 4 migration, and it seems like it impacts a lot of developer.

I read all the discussion and maybe I miss-understand but I feel like there is currently no solution to avoid this extra sql in every diff. Isn't it ?

The comment was really useful for custom type. If there is no known solution, what about re-introducing it back ? At least this could be optional like an option to toggle. This way if comments have bad impact for some user they can disable it.

uncaught · 2024-10-08T19:12:15Z

As I said, we have had a solution for years and it kept working with dbal 4.

I would create a PR, but it's for mysql only 🫤

meiyasan · 2024-10-08T22:01:00Z

I encounter the same issue with my custom types.

I keep generating dozen of lines like
$this->addSql('ALTER TABLE activity_events CHANGE status status ENUM(\'DISPATCHED\', \'ONGOING\', \'SUCCEEDED\', \'FAILED\') NOT NULL');
in my migration diff, when it worked fine with COMMENT \'(DC2Type:enumActivityEventStatus)\''

It does not give a good experience with the DBAL 4 migration, and it seems like it impacts a lot of developer.

I read all the discussion and maybe I miss-understand but I feel like there is currently no solution to avoid this extra sql in every diff. Isn't it ?

The comment was really useful for custom type. If there is no known solution, what about re-introducing it back ? At least this could be optional like an option to toggle. This way if comments have bad impact for some user they can disable it.

@VincentLanglet Your feeling is correct, in my opinion. I haven’t found any alternatives either. I’ve been begging for merging this PR doctrine/dbal#6444 for months, but they don’t want to for obscure reasons.
While this doesn’t have to be a permanent solution, removing those lines too early feels like a quick shot. They could just leave them in place until a more mature alternative is found.

A good legacy solution would be to follow the approach of the doctrine/annotation bundle and make it optional, but they seem to have missed the point for months.

If you’re interested, I created this package - a code modifier that reintroduces the missing lines: https://packagist.org/packages/glitchr/doctrine-dc2type. Please use with caution and inspect the package on a fresh symfony skeleton if you wish, there is nothing too complex or fancy, but it is nothing official.
This is the only alternative I had for the sake of my projects.

derrabus · 2024-10-08T23:55:04Z

Your complaints have been duly noted. Now, can we please get back to discussing solutions?

I mean, you can just stay on DBAL 3 until the problem is solved or help working on a proper solution. We will still maintain DBAL 3 for some time.

VincentLanglet · 2024-10-09T07:26:28Z

Now, can we please get back to discussing solutions?

Sure, but I'm unsure how to help. I certainly lack of knowledge about how this work.

If I debug MySQLSchemaManager::_getPortableTableColumnDefinition, I feel like the logic here is incorrect
https://github.com/doctrine/dbal/blob/7a8252418689feb860ea8dfeab66d64a56a64df8/src/Schema/MySQLSchemaManager.php#L118-L135

When the data_type in my DB is enum('DISPATCHED','ONGOING','SUCCEEDED','FAILED')

$dbType = strtok($dbType, '(), ') is enum
$length = $tableColumn['length'] ?? strtok('(), '); is 'dispatched

Personally, I could almost fix my issue if I overriden the

public function getMappedDatabaseTypes(AbstractPlatform $platform): array

method in my customs Type, but it would require to update the MySQLSchemaManager to something like

        $dbType = strtolower($tableColumn['type']);
        if (!str_starts_with($dbType, 'enum(')) {
            $dbType = strtok($dbType, '(), ');
            assert(is_string($dbType));

            $length = $tableColumn['length'] ?? strtok('(), ');
        } else {
            $length = null;
        }

Then https://www.doctrine-project.org/projects/doctrine-orm/en/latest/cookbook/mysql-enums.html could be updated to implement

public function getMappedDatabaseTypes(AbstractPlatform $platform): array
    {
        $values = array_map(function($val) { return "'".$val."'"; }, $this->values);

        return ["ENUM(".implode(",", $values).")"];
    }

(the abstract EnumType could even be provided by doctrine eventually...)

Do you think it could a good improvement @greg0ire @derrabus ? (And how avoid a BC break...)

I mean, you can just stay on DBAL 3 until the problem is solved or help working on a proper solution. We will still maintain DBAL 3 for some time.

DBAL 4 still have benefits ; I like the evolution done to bigint.

michnovka · 2024-10-09T09:53:51Z

Should the VerbatimType live inside Migration or DBAL? I would argue Migration, since it should not be used anywhere else.

And @stof

as long as AbstractSchemaManager::getPortableTableColumnDefinition does not allow to access the original definition for enum fields during introspection, this verbatim type won't solve it.

This is not true, we have that, see #1441 (comment)

We can just create a VerbatimType from within here and just give it this $tableColumn property. We cannot use TypeRegistry for this case, as every VerbatimType will be its own instance.

But I think we can handle all this inside SchemaManager's _getPortableTableColumnDefinition(). Seems well have to edit every single one tho, which overrides this method and introduce this logic.

Then we just make special case in comparator for VerbatimType and we're golden.

Am I missing something, or does it sound too easy to you too?

derrabus · 2024-10-09T16:04:30Z

Does doctrine/dbal#6536 solve the issue for those of you who are using enum columns?

stof · 2024-10-09T16:18:28Z

@michnovka both the schema introspection and the schema comparison are provided by DBAL. So it would definitely be used outside Migrations (and Migrations would not even use it directly but only through the usages in DBAL itself)

VincentLanglet · 2024-10-09T16:27:26Z

Does doctrine/dbal#6536 solve the issue for those of you who are using enum columns?

I'm getting No changes detected in your mapping information. with your branch when I run do:mi:diff.
That's an error message I like.

Also I tried to change my mapping and/or change my enumType ; in both case the diff is correct.
It's awesome, thanks a lot @derrabus

| Q | A |------------- | ----------- | Type | feature | Fixed issues | doctrine/migrations#1441 (partly) #### Summary This PR adds an `EnumType` that allows us to introspect and diff tables that make use of MySQL's `ENUM` column type.

michnovka · 2024-10-11T12:36:36Z

Thanks @derrabus this indeed solves the issues with ENUM and migrations.

Only thing I noticed was that if my sqlDeclaration defined the type as lowercase enum('a','b') it would still cause migrations (where up would have ENUM('a','b') and down enum('a','b'). Is this expected?

Changing my custom type's sql declaration to uppercase ENUM resolved this.

derrabus · 2024-10-11T12:49:37Z

So, this is your fix then. 🙂

PowerKiKi · 2024-10-16T10:11:17Z

@derrabus (or @michnovka ?), could you please clarify how the new EnumType should be used ?

I updated my playground to the latest versions of all packages, and I still see varchar in my DB, instead of the expected enum. I thought it should work out of the box. Did I misunderstand something ?

The model is declared there, and Doctrine is configured in the simplest manner I know of.

Basically I am doing:

<?php

use Doctrine\ORM\Mapping as ORM;

enum UserRole: string
{
    case Visitor = 'visitor';
    case Member = 'member';
    case Admin = 'admin';
}


enum UserStatus: string
{
    case New = 'new';
    case Active = 'active';
    case Archived = 'archived';
}

#[ORM\Entity]
class User
{
    #[ORM\Id]
    #[ORM\Column(type: 'integer')]
    public ?int $id = null;

    #[ORM\Column]
    public UserRole $role = UserRole::Visitor;

    #[ORM\Column]
    public UserStatus $status = UserStatus::New;
}

And that:

function createEntityManager(): EntityManager
{
    $config = ORMSetup::createAttributeMetadataConfiguration(['my-path-to-my-entity'], true);

    $connection = DriverManager::getConnection([
        'driverClass' => Driver::class,
        'host' => '127.0.0.1',
        'dbname' => 'test',
        'user' => 'test',
        'password' => 'test',
    ], $config);

    $entityManager = new EntityManager($connection, $config);

    return $entityManager;
}

...but it still fails to have enum in my DB.

michnovka · 2024-10-16T11:48:08Z

@PowerKiKi still by default doctrine will use StringType for your enum. You will need to customize

    #[ORM\Column(type: 'enum', enumType: UserRole::class, values: [...])]
    public UserRole $role = UserRole::Visitor;

If you do not want to specify values here, and want by default all enum values to be used in DB as well, you have to use a custom type.

You can look into https://github.com/Elao/PhpEnums that makes this much easier.

You can also look into Elao/PhpEnums#212 where I show on top my implementation which auto-registers enums as types.

derrabus · 2024-10-16T12:11:44Z

@derrabus (or @michnovka ?), could you please clarify how the new EnumType should be used ?

Please read doctrine/orm#11666.

If you do not want to specify values here, and want by default all enum values to be used in DB as well, you have to use a custom type.

No, don't do that. Custom enum types should be obsolete now.

You can look into https://github.com/Elao/PhpEnums that makes this much easier.

Not anymore, really.

michnovka · 2024-10-16T15:49:20Z

@derrabus this is so cool, thanks a lot, I was unaware of this new PR

@PowerKiKi if you still want to use MySQL enums by default, you can take advantage of TypedFieldMapper. Here is my solution for the interested:

create custom EnumTypedFieldMapper

<?php

declare(strict_types=1);

namespace App\Doctrine\ORM\TypedFieldMapper;

use Doctrine\DBAL\Types\Types;
use Doctrine\ORM\Mapping\TypedFieldMapper;
use ReflectionNamedType;
use ReflectionProperty;

class EnumTypedFieldMapper implements TypedFieldMapper
{
    /**
     * {@inheritdoc}
     */
    public function validateAndComplete(array $mapping, ReflectionProperty $field): array
    {
        $type = $field->getType();

        if (
            ! isset($mapping['type'])
            && ($type instanceof ReflectionNamedType)
        ) {
            if (!$type->isBuiltin() && enum_exists($type->getName())) {
                $mapping['type'] = Types::ENUM;
            }
        }

        return $mapping;
    }
}

adjust services.yaml to register this service and also pass it into the ChainTypedFieldMapper together with the DefaultTypedFieldMapper (note that in my example you can even see how I map some other typed fields to DBAL types)

services:
    App\Doctrine\ORM\TypedFieldMapper\EnumTypedFieldMapper:

    Doctrine\ORM\Mapping\DefaultTypedFieldMapper:
        arguments:
            $typedFieldMappings:
                Decimal\Decimal: 'decimal'
                Carbon\CarbonImmutable: 'datetime_immutable'

    Doctrine\ORM\Mapping\ChainTypedFieldMapper:
        arguments:
            $typedFieldMappers:
                - '@App\Doctrine\ORM\TypedFieldMapper\EnumTypedFieldMapper'
                - '@Doctrine\ORM\Mapping\DefaultTypedFieldMapper'

then replace the default DefaultTypedFieldMapper with your newly configured ChainTypedFieldMapper inside doctrine.yaml - note that the chain makes sure that the default mapper is still used, just executed AFTER your custom enum mapper

doctrine:
    orm:
        typed_field_mapper: 'Doctrine\ORM\Mapping\ChainTypedFieldMapper'

now you can use

#[ORM\Column]
public UserRole $role = UserRole::Visitor;

And it will make your field an ENUM type in database

Note that this requires ORM 3.3, DBAL 4.2 and DoctrineBundle 2.13

PowerKiKi · 2024-10-16T16:46:34Z

@derrabus, this is great, thank you !

My playground is now working as expected. Specifying the type of the column is an obvious solution I should have thought about, #[ORM\Column(type: 'enum')] !

@michnovka, thanks for the hint for ChainTypedFieldMapper. It is a nice to have that I might implement at some point.

From my point of view, which is only concerned about enum, this issue may be closed as resolved.

acasademont · 2024-10-28T18:29:57Z

Well, we still have the issue with the DATETIME(6) as @WubbleWobble pointed out in #1441 (comment)

Not sure if this should remain open as a "placeholder" for that problem too.

A short patch that adds a middleware to the connection setting a custom platform (extended on MysqlPlatform80) and modifying the sql snippet to take into account the length. This now generates empty diffs for us, hope it might help someone else.

use Doctrine\DBAL\Driver;
use Doctrine\DBAL\Driver\Middleware;
use Doctrine\DBAL\Platforms\AbstractPlatform;
use Doctrine\DBAL\Platforms\MySQL80Platform;
use Doctrine\DBAL\ServerVersionProvider;

class LengthAwareMysqlMiddleware implements Middleware
{
    public function wrap(Driver $driver): Driver
    {
        return new class($driver) extends Middleware\AbstractDriverMiddleware {
            public function getDatabasePlatform(ServerVersionProvider $versionProvider): AbstractPlatform
            {
                return new class extends MySQL80Platform {
                    public function getDateTimeTypeDeclarationSQL(array $column): string
                    {
                        $sql = parent::getDateTimeTypeDeclarationSQL($column);

                        if (isset($column['length']) && $column['length'] > 0) {
                            $sql .= '(' . $column['length'] . ')';
                        }

                        return $sql;
                    }
                };
            }
        };
    }
}

derrabus · 2024-10-28T19:29:20Z

Well, we still have the issue with the DATETIME(6) as @WubbleWobble pointed out in #1441 (comment)

… and my answer to his comment still applies.

tomsykes · 2024-10-29T08:25:23Z

I still generally have a problem with the dropping of D2Ctype comments. By doing so. Doctrine has lost the ability to have more than one DBAL type having the same column definition.

When the comparator tries to detect if anything has changed, it's using the column definition to determine the DBAL type. When there are two DBAL types with the same column definition, it just picks the first one. When there were D2Ctype comments, the comparator could distinguish between the two DBAL types.

uncaught · 2024-10-29T10:05:05Z

When the comparator tries to detect if anything has changed, it's using the column definition to determine the DBAL type.

It does? I can't imagine this is the case. It would have to run through all possible parameters, too, to find the type based on a schema. Just imagine it finds a decimal(42,7) - how would that deterministically be resolved to a DBAL type?

All the migrations need to do is compare what is with what should be. And for that the comments were superfluous.

tomsykes · 2024-10-29T10:18:40Z

I think you're right @uncaught, you would expect the comparator to be generating a dummy schema based on the current state of the entities, and then compare that with the actual schema, but that's not what's happening.

@michnovka details how it works in their first comment

Notice the call to $platform->getDoctrineTypeMapping($dbType). It's resolving a doctrine type based on the database definition.

Edit: lets be more precise about this... the comparator isn't generating anything, it's just comparing what it's being given... but the point is the the AbstractSchemaManager + [PlatformSpecific]SchemaManager are providing the comparator with column definitions that are based on a resolved DBAL type, and not the actual underlying SQL type, so when it comes to the comparison it's not necessarily a fair representation of what's actually in the database - hence why the migrations are being continually generated.

michnovka mentioned this issue Jul 7, 2024

Add ENUM type into DBAL 4 doctrine/dbal#6465

Closed

berkut1 mentioned this issue Jul 8, 2024

Type Mapping Discrepancies Between DBAL3 and DBAL4 doctrine/dbal#6466

Open

greg0ire mentioned this issue Sep 6, 2024

Generated migrations do not consider columnDefinition=... clause doctrine/dbal#6519

Open

derrabus mentioned this issue Oct 9, 2024

Implement an EnumType for MySQL/MariaDB doctrine/dbal#6536

Merged

michnovka mentioned this issue Oct 10, 2024

Bump DBAL 4 min version to 4.2.0 Elao/PhpEnums#234

Closed

This was referenced Nov 27, 2024

Custom datetime with support of microseconds not properly works after update of dbal from 3.6 to 4.2.1 doctrine/dbal#6631

Open

Custom datetime with support of microseconds always generate unnecessary migration after update of dbal from 3.6 to 4.2.1 #1472

Closed

Migrations always generated for custom type with DBAL 4 #1441

Migrations always generated for custom type with DBAL 4 #1441

Comments

michnovka commented Jul 3, 2024 • edited Loading

Bug Report

Summary

michnovka commented Jul 3, 2024

greg0ire commented Jul 3, 2024 • edited Loading

michnovka commented Jul 3, 2024 • edited Loading

berkut1 commented Jul 3, 2024 • edited Loading

michnovka commented Jul 3, 2024

berkut1 commented Jul 3, 2024 • edited Loading

michnovka commented Jul 3, 2024 • edited Loading

berkut1 commented Jul 3, 2024 • edited Loading

michnovka commented Jul 3, 2024

berkut1 commented Jul 3, 2024 • edited Loading

michnovka commented Jul 3, 2024 • edited Loading

greg0ire commented Jul 3, 2024

michnovka commented Jul 3, 2024 • edited Loading

greg0ire commented Jul 3, 2024

michnovka commented Jul 3, 2024

berkut1 commented Jul 3, 2024 • edited Loading

greg0ire commented Jul 3, 2024

michnovka commented Jul 3, 2024

berkut1 commented Jul 3, 2024 • edited Loading

michnovka commented Jul 4, 2024

greg0ire commented Jul 4, 2024

michnovka commented Jul 4, 2024

greg0ire commented Jul 4, 2024

michnovka commented Jul 5, 2024

greg0ire commented Jul 5, 2024

berkut1 commented Jul 6, 2024

michnovka commented Jul 7, 2024 • edited Loading

berkut1 commented Jul 7, 2024 • edited Loading

michnovka commented Sep 28, 2024

PowerKiKi commented Sep 28, 2024

morozov commented Oct 8, 2024

VincentLanglet commented Oct 8, 2024

uncaught commented Oct 8, 2024

meiyasan commented Oct 8, 2024 • edited Loading

derrabus commented Oct 8, 2024

VincentLanglet commented Oct 9, 2024 • edited Loading

michnovka commented Oct 9, 2024 • edited Loading

derrabus commented Oct 9, 2024

stof commented Oct 9, 2024

VincentLanglet commented Oct 9, 2024

michnovka commented Oct 11, 2024

derrabus commented Oct 11, 2024

PowerKiKi commented Oct 16, 2024

michnovka commented Oct 16, 2024

derrabus commented Oct 16, 2024

michnovka commented Oct 16, 2024

PowerKiKi commented Oct 16, 2024

acasademont commented Oct 28, 2024

derrabus commented Oct 28, 2024

tomsykes commented Oct 29, 2024

uncaught commented Oct 29, 2024

tomsykes commented Oct 29, 2024 • edited Loading

michnovka commented Jul 3, 2024 •

edited

Loading

greg0ire commented Jul 3, 2024 •

edited

Loading

michnovka commented Jul 3, 2024 •

edited

Loading

berkut1 commented Jul 3, 2024 •

edited

Loading

berkut1 commented Jul 3, 2024 •

edited

Loading

michnovka commented Jul 3, 2024 •

edited

Loading

berkut1 commented Jul 3, 2024 •

edited

Loading

berkut1 commented Jul 3, 2024 •

edited

Loading

michnovka commented Jul 3, 2024 •

edited

Loading

michnovka commented Jul 3, 2024 •

edited

Loading

berkut1 commented Jul 3, 2024 •

edited

Loading

berkut1 commented Jul 3, 2024 •

edited

Loading

michnovka commented Jul 7, 2024 •

edited

Loading

berkut1 commented Jul 7, 2024 •

edited

Loading

meiyasan commented Oct 8, 2024 •

edited

Loading

VincentLanglet commented Oct 9, 2024 •

edited

Loading

michnovka commented Oct 9, 2024 •

edited

Loading

tomsykes commented Oct 29, 2024 •

edited

Loading