[BUG] Cython 3.0.0 release breaking flytekit editable install setup

[Benjamintdk]

Describe the bug

Getting a cythonsources error when I follow this set of instructions to set up my environment and run make setup.

Collecting pyyaml<6 (from -r requirements.in (line 8))
  Downloading PyYAML-5.4.1.tar.gz (175 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 175.1/175.1 kB 6.1 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [68 lines of output]
      /private/var/folders/wv/3lvjxhys2mz8yg_tspn408lm0000gp/T/pip-build-env-l2yb7365/overlay/lib/python3.11/site-packages/setuptools/config/setupcfg.py:293: _DeprecatedConfig: Deprecated config in `setup.cfg`
      !!
      
              ********************************************************************************
              The license_file parameter is deprecated, use license_files instead.
      
              By 2023-Oct-30, you need to update your project and remove deprecated calls
              or your builds will no longer be supported.
      
              See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
              ********************************************************************************
      
      !!
        parsed = self.parsers.get(option_name, lambda x: x)(value)
      running egg_info
      writing lib3/PyYAML.egg-info/PKG-INFO
      writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt
      writing top-level names to lib3/PyYAML.egg-info/top_level.txt
      Traceback (most recent call last):
        File "/Users/benjamin.chew/.virtualenvs/flytekit/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/Users/benjamin.chew/.virtualenvs/flytekit/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/benjamin.chew/.virtualenvs/flytekit/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/wv/3lvjxhys2mz8yg_tspn408lm0000gp/T/pip-build-env-l2yb7365/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 341, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/wv/3lvjxhys2mz8yg_tspn408lm0000gp/T/pip-build-env-l2yb7365/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 323, in _get_build_requires
          self.run_setup()
        File "/private/var/folders/wv/3lvjxhys2mz8yg_tspn408lm0000gp/T/pip-build-env-l2yb7365/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 338, in run_setup
          exec(code, locals())
        File "<string>", line 271, in <module>
        File "/private/var/folders/wv/3lvjxhys2mz8yg_tspn408lm0000gp/T/pip-build-env-l2yb7365/overlay/lib/python3.11/site-packages/setuptools/__init__.py", line 107, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/wv/3lvjxhys2mz8yg_tspn408lm0000gp/T/pip-build-env-l2yb7365/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
                 ^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/wv/3lvjxhys2mz8yg_tspn408lm0000gp/T/pip-build-env-l2yb7365/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/private/var/folders/wv/3lvjxhys2mz8yg_tspn408lm0000gp/T/pip-build-env-l2yb7365/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/private/var/folders/wv/3lvjxhys2mz8yg_tspn408lm0000gp/T/pip-build-env-l2yb7365/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 1234, in run_command
          super().run_command(command)
        File "/private/var/folders/wv/3lvjxhys2mz8yg_tspn408lm0000gp/T/pip-build-env-l2yb7365/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/private/var/folders/wv/3lvjxhys2mz8yg_tspn408lm0000gp/T/pip-build-env-l2yb7365/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 314, in run
          self.find_sources()
        File "/private/var/folders/wv/3lvjxhys2mz8yg_tspn408lm0000gp/T/pip-build-env-l2yb7365/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 322, in find_sources
          mm.run()
        File "/private/var/folders/wv/3lvjxhys2mz8yg_tspn408lm0000gp/T/pip-build-env-l2yb7365/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 551, in run
          self.add_defaults()
        File "/private/var/folders/wv/3lvjxhys2mz8yg_tspn408lm0000gp/T/pip-build-env-l2yb7365/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 589, in add_defaults
          sdist.add_defaults(self)
        File "/private/var/folders/wv/3lvjxhys2mz8yg_tspn408lm0000gp/T/pip-build-env-l2yb7365/overlay/lib/python3.11/site-packages/setuptools/command/sdist.py", line 104, in add_defaults
          super().add_defaults()
        File "/private/var/folders/wv/3lvjxhys2mz8yg_tspn408lm0000gp/T/pip-build-env-l2yb7365/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/sdist.py", line 251, in add_defaults
          self._add_defaults_ext()
        File "/private/var/folders/wv/3lvjxhys2mz8yg_tspn408lm0000gp/T/pip-build-env-l2yb7365/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/sdist.py", line 336, in _add_defaults_ext
          self.filelist.extend(build_ext.get_source_files())
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "<string>", line 201, in get_source_files
        File "/private/var/folders/wv/3lvjxhys2mz8yg_tspn408lm0000gp/T/pip-build-env-l2yb7365/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
          raise AttributeError(attr)
      AttributeError: cython_sources
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

It seems that the recent version of Cython 3.0.0 that was released just yesterday was causing pyyaml 5.4.2 to break. It also seems that pyyaml's version has been constrained to under 6 to accommodate for docker compose, as per this issue.

Expected behavior

Running make setup should not throw an error and the installation should complete successfully.

Additional context to reproduce

No response

Screenshots

No response

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[eapolinario]

@Benjamintdk , thanks for opening the issue! I'm following up on it on flyteorg/flytekit#1752.

[eapolinario]

@Benjamintdk , can you confirm which combination of python version and architecture you're seeing this?

[Benjamintdk]

Hi @eapolinario, am currently using Python 3.11.3 and MacOS.

[eapolinario]

Can you rebase? If this is for local development only, then flyteorg/flytekit#1746 should give you enough room to start developing locally.

[Benjamintdk]

I managed to get around this temporarily by downgrading pyyaml to 5.3.1 instead, but this has security vulnerabilities, so will definitely look for other alternatives moving forward. Would be happy to have a discussion and work on a more permanent fix for this.

[eapolinario]

Closing this as we have a workaround in place.

It's a bit unfortunate that for older versions of python we end up installing a version of pyyaml that's affected by a serious CVE, but unless that project decides to handle it there isn't a lot that flytekit can do about it.