Rework the continuous integration jobs

[greg0ire]

This is an attempt at implementing #3347 (comment)

• For the latest stable PHP version (which is currently 8.3), test all drivers with all platform versions.
• All other supported PHP versions (7.4, 8.2, nightly), test each driver with the latest version of the platform.

This results in a slight reduction of the number of jobs (from 92 to 85)

[morozov]

Thanks @greg0ire for taking a stab at this. Hopefully, this number will drop even further when this cleanup is applied to newer major DBAL versions where we support fewer versions of PHP and database platforms.

The next thing that stands out is the number of distinct database platform versions we test against. For example, testing against 9 (!) different MariaDB versions looks wasteful.

Maybe we could assume certain risks and test only the oldest and newest versions of at least some platforms? Even though technically possible, it's quite unlikely that a given issue happens in a specific platform version but not in the oldest or newest. We can also drive this optimization by test coverage. If there is code for a specific platform, we want it covered. Otherwise, we assume that the platform behaves the same as others.

A similar pattern may be applied to testing SQLite. We test it against all supported minor versions of PHP but could test only against the most significant ones (e.g. each latest major + nightly).

[greg0ire]

Down to 65% but Codecov, which seems to finally work properly mentions a slight drop in code coverage:

Let me try to restore just what's necessary.

[greg0ire]

Restored some builds according to Codecov, and according to version_compare calls I found for MariaDB and PostgreSQL, that were not picked up by Codecov for some reason.

[greg0ire]

72 checks

[greg0ire]

There is still a slight drop in coverage, and I don't understand it, the 10.5 mariadb job should cover it.

[morozov]

Codecov has never calculated coverage accurately and consistently, so if some code is executed but isn't reported as such, I wouldn't bother pleasing Codecov. I see the latest build showing -0.01% code coverage difference, which is fine (whatever that means).

[greg0ire]

whatever that means

This is what it means:

I believe that line should be covered, but when I download and unzip https://github.com/doctrine/dbal/actions/runs/11908444501/artifacts/2205889060, which corresponds to phpunit-mariadb-10.5-pdo_mysql-8.3.coverage , which I would expect to cover this

I get

      <line num="39" type="method" name="createDatabasePlatformForVersion" visibility="public" complexity="13" crap="13.37" count="88"/>
      <line num="41" type="stmt" count="88"/>
      <line num="43" type="stmt" count="88"/>
      <line num="44" type="stmt" count="62"/>
      <line num="45" type="stmt" count="62"/>
      <line num="46" type="stmt" count="2"/>
      <line num="49" type="stmt" count="60"/>
      <line num="50" type="stmt" count="4"/>
      <line num="53" type="stmt" count="56"/>
      <line num="54" type="stmt" count="42"/>
      <line num="57" type="stmt" count="14"/>
      <line num="58" type="stmt" count="0"/>
      <line num="61" type="stmt" count="14"/>
      <line num="62" type="stmt" count="14"/>
      <line num="63" type="stmt" count="14"/>
      <line num="64" type="stmt" count="14"/>
      <line num="65" type="stmt" count="14"/>
      <line num="66" type="stmt" count="14"/>
      <line num="68" type="stmt" count="14"/>
      <line num="69" type="stmt" count="6"/>
      <line num="72" type="stmt" count="26"/>
      <line num="74" type="stmt" count="24"/>
      <line num="75" type="stmt" count="4"/>
      <line num="76" type="stmt" count="0"/>
      <line num="77" type="stmt" count="0"/>
      <line num="78" type="stmt" count="0"/>
      <line num="79" type="stmt" count="0"/>
      <line num="80" type="stmt" count="0"/>
      <line num="81" type="stmt" count="0"/>
      <line num="84" type="stmt" count="4"/>
      <line num="87" type="stmt" count="20"/>
      <line num="88" type="stmt" count="6"/>
      <line num="89" type="stmt" count="4"/>
      <line num="90" type="stmt" count="4"/>
      <line num="91" type="stmt" count="4"/>
      <line num="92" type="stmt" count="4"/>
      <line num="93" type="stmt" count="4"/>
      <line num="94" type="stmt" count="4"/>
      <line num="97" type="stmt" count="6"/>
      <line num="100" type="stmt" count="14"/>
      <line num="101" type="stmt" count="8"/>
      <line num="102" type="stmt" count="2"/>
      <line num="103" type="stmt" count="2"/>
      <line num="104" type="stmt" count="2"/>
      <line num="105" type="stmt" count="2"/>
      <line num="106" type="stmt" count="2"/>
      <line num="107" type="stmt" count="2"/>
      <line num="110" type="stmt" count="8"/>
      <line num="113" type="stmt" count="6"/>
      <line num="114" type="stmt" count="6"/>
      <line num="115" type="stmt" count="6"/>
      <line num="116" type="stmt" count="6"/>
      <line num="117" type="stmt" count="6"/>
      <line num="118" type="stmt" count="6"/>
      <line num="121" type="stmt" count="14"/>

Line 58 is not covered although it should, line 61 is covered although it shouldn't. In fact, many lines that don't have to do with this version range are covered. I think that is because of

dbal/tests/Driver/VersionAwarePlatformDriverTest.php

Lines 39 to 52 in 1a30708

	return [
	['5.7.0', MySQLPlatform::class],
	['8.0.11', MySQL80Platform::class],
	['8.4.1', MySQL84Platform::class],
	['9.0.0', MySQL84Platform::class],
	['5.5.40-MariaDB-1~wheezy', MariaDBPlatform::class],
	['5.5.5-MariaDB-10.2.8+maria~xenial-log', MariaDBPlatform::class],
	['10.2.8-MariaDB-10.2.8+maria~xenial-log', MariaDBPlatform::class],
	['10.2.8-MariaDB-1~lenny-log', MariaDBPlatform::class],
	['10.5.2-MariaDB-1~lenny-log', MariaDB1052Platform::class],
	['10.6.0-MariaDB-1~lenny-log', MariaDB1060Platform::class],
	['10.9.3-MariaDB-1~lenny-log', MariaDB1060Platform::class],
	['11.0.2-MariaDB-1:11.0.2+maria~ubu2204', MariaDB1010Platform::class],
	];

[greg0ire]

Ah I think I get it. 10.5 probably resolves to the latest 10.5, which is handled by another case. Let me add another version.

[greg0ire]

@morozov that worked I must have been looking at the wrong version of AbstractMySQLDriver yesterday.

Now we are up to 75 CI checks, and there is no coverage drop.

Codecov has never calculated coverage accurately and consistently,

I will try to pay more attention to this now that #6611 is merged, in case the issues you notice are caused by something else.

[morozov]

What would be a good place where we could document the rules for building the matrix? Ideally, it should be possible to maintain it and reference every time we make a change in the matrix. But also it shouldn't be under VCS because it's internal knowledge, and I wouldn't like to have to go through the PR and code review process to update it.

[morozov]

in case the issues you notice are caused by something else.

What I meant is the cases like this:

There was no ongoing 4.2 build at the time of making this screenshot but the reported coverage was abnormally low.

[greg0ire]

That's because it only received the AppVeyor reports https://app.codecov.io/gh/doctrine/dbal/commit/db6428c4b12c44239e5716ec5cc0fc705ac1c072

Presumably because of codecov/codecov-action#1633, which I'm hopeful is fixed. I'm going to merge up the upgrade, so that we have v5 everywhere.

[greg0ire]

Done, and that README looks better already 🙂

[greg0ire]

What would be a good place where we could document the rules for building the matrix? Ideally, it should be possible to maintain it and reference every time we make a change in the matrix. But also it shouldn't be under VCS because it's internal knowledge, and I wouldn't like to have to go through the PR and code review process to update it.

A pinned issue then? Like doctrine/orm#11208 ?

[morozov]

A pinned issue then?

Sounds good. I'll file one some time.

UPD: #6620