About Rope's type hierarchy

[edreamleo]

@lieryan This discussion is background for PR #677: use single inheritance.

The following table describes the present type hierarchy for the classes in the builtins, pyobjects, and pyobjectsdef modules:

class AbstractClass(PyObject):
class AbstractFunction(PyObject):
class AbstractModule(PyObject):

class BuiltinClass(_BuiltinElement, pyobjects.AbstractClass):
class BuiltinFunction(_BuiltinElement, pyobjects.AbstractFunction):
class BuiltinModule(pyobjects.AbstractModule):
class Generator(pyobjects.AbstractClass):
class Iterator(pyobjects.AbstractClass):
class Lambda(pyobjects.AbstractFunction):

class PyObject:
class PyDefinedObject:

class PyClass(PyDefinedObject, AbstractClass):
class PyFunction(PyDefinedObject, AbstractFunction):
class _PyModule(PyDefinedObject, AbstractModule):

The PyObject and PyDefinedObject classes appear to be unrelated, but let's do some inference on this table!

1. All Abstract classes are subclasses of PyObject.
2. All classes in the builtins module are subclasses of the Abstract classes.
3. All subclasses of PyDefinedObject objects are subclasses of Abstract objects.

Therefore: all of the above classes are effectively subclasses of PyObject.

What is PyDefinedObject meant to be?

Rope's present code proclaims that PyDefinedObject is not a subclass of PyObject. As a recent newbie, I found PyDefinedObjects apparent lack of a base class bewildering.

Certainly, the PyObject and PyDefinedObject are not the same class. Imo, both should have docstrings. These docstrings don't necessarily have to be complete—they might link (via url's) to longer discussions in the Theory of Operation.

What about mypy annotations?

Because all PyDefinedObjects are PyObjects the annotation (in Rope's existing code, remember):

Union[PyObject, PyDefinedObject]

is equivalent to just:

PyObject

In other words, unless there are bugs (gaps in mypy's inference) mypy can not possibly infer anything more about the Union than it can about PyObject

This observation brings me back to the previous section. I wonder whether LieRyan is somehow using the Union annotation as a form of comment. If so, the proper (imo) thing to do is to use the simpler annotation and add new documentation to docstrings and the Theory of Operation.

Summary

Rope's present class hierarchy proclaims that PyDefinedObject has no base class. Newbies will (should!) find this lack of a base class confusing.

In fact, PyDefinedObject is a subclass of PyObject. This relationship arises implicitly from the table given above. PR #677 makes that relationship explicit.

In Rope's present codebase, mypy must treat the annotations Union[PyObject, PyDefinedObject] and PyObject as equivalent. Handling these two annotations differently would be a bug. In any event, the PyObject and PyDefinedObject classes should have docstrings.

I conjecture that LieRyan prefers the Union annotation because it acts like a comment, but I have no idea what that comment could be!

[lieryan]

I disagree that this is confusing to newbies. It's only confusing to people who come from a different language that has a different expectation of how classes and inheritances should work and doesn't actually want to learn Python. Actual beginners don't have an incorrect expectation that they had to unlearn.

I think the way you are describing this lends too much credence to the lexical (=as written in file) heritage of a class. But in Python, the lexical scope of a class does not actually exist; the only thing that actually matters is the runtime heritage. It's a disservice to teach beginners into expecting that lexical heritage should matter at all in Python.

In Python, there's no expectation that methods should only call methods that exists in the lexical parent of that class, as the lexical heritage is a construction that only exists in some static languages, but Python is a dynamic language, the lexical class heritage is (and should be!!) irrelevant. The only class heritage that matters is what is the runtime type of the object and its MRO, the lexical heritage is just a distraction that hinders actual understanding of Python and a piece of Python code.

Usually, the way Python are taught is that you'd be taught to explore unfamiliar objects in the REPL. People don't normally go around reading random classes like PyDefinedObject to understand what it does, the typical codebase of any non-trivial project is too large for that strategy to ever be fruitful. Instead, you usually would start with an instance of the object that you're working on, for example, in myobj you might have an instance of pynamesdef.DefinedName and then you start exploring with that:

>>> myobj
<rope.base.pynamesdef.DefinedName at 0x7fcb90255ae0>

and from that point on, you find out what methods will get called by get_module() by using the dir() function to find out what you can do with it:

>>> dir(myobj)
[ 
  ...
  'get_definition_location',
  'get_object',
  'pyobject'
]
>>> myobj.get_object
<bound method DefinedName.get_object of <rope.base.pynamesdef.DefinedName object at 0x7fcb90255ae0>>

>>> myobj.get_object()
<rope.base.pyobjectsdef.PyClass "rope.base.project::Project" at 0x7fcb90255990>
>>> myobj.get_object().get_module   # not the same `get_module` method as above!!
<bound method PyDefinedObject.get_module of <rope.base.pyobjectsdef.PyClass "rope.base.project::Project" at 0x7fcb90255990>>

And then when an object may be implemented at multiple levels, you use the mro to understand what gets called and in what order:

>>> myobj.get_object()
<rope.base.pyobjectsdef.PyClass "rope.base.project::Project" at 0x7fcb90255990>
>>> type(myobj.get_object()).__mro__
(rope.base.pyobjectsdef.PyClass,
 rope.base.pyobjects.PyClass,
 rope.base.pyobjects.PyDefinedObject,
 rope.base.pyobjects.AbstractClass,
 rope.base.pyobjects.PyObject,
 object)

>>> import inspect  # and let's explore the source code
>>> print(inspect.getsource(obj.get_object().get_module))
    def get_module(self):
        current_object = self
        while current_object.parent is not None:
            current_object = current_object.parent
        return current_object

Once you become more advanced learner, and use a more powerful REPL like IPython, most of the things above would feel second nature, and even just be much simpler using the autocomplete mechanism which does all that for you, or the ?? magic operator.

Unless one has an incorrect mental model of how Python works, under no circumstances, would the lack of get_module() method in the base classes of PyClass should ever cause a practical problem of understanding what gets called when. You'd expect get_module() to exist somewhere in the inheritance tree of PyClass, and other than for constructing the MRO of PyClass, the lexical heritage of intermediate classes like PyDefinedObject or AbstractClass itself are just irrelevant.

IMO, the lack of lexical heritage scope actually is what makes Python much easier to understand for beginners. You no longer have conflicts between lexical parentage of an object and the actual code flow, nor the issues that would often arise from that that often happens when using a static language. IMO, it's one of the reason that makes Python a simple to use language, it's a practical language with a very simple and dumb execution model that doesn't burden you with many theoretical baggages.

[edreamleo]

@lieryan Thanks for your reply. I'll consider it carefully.

[edreamleo]

@lieryan It's time to end my participation in the Rope project. We disagree on too much.

I have closed all my issues and PRs, archived ekr-fork-rope2, and will make no further contributions.

Thanks for the help you have given me. The experience has been worthwhile.

[lieryan]

Hi @edreamleo, this comes as a bit of a surprise, I'm saddened that things didn't work out, I really wanted to make this work though I understand your perspective.

Thank you for participating. We don't always agree but I always appreciate what you had done, whether or not the contributions get merged, your contributions has been very great and thought provoking. So, I have to reiterate this again: thank you very much.

Our conversations have taught me a lot as much as well, and I had to thank you for that as well. I too am still figuring out how to run a project like this, so I realise I am not always the easiest to work with, so I would like to apologise if there is anything I had said or did anything that was wrong.

The door will always be open here, you will always be welcome at any time. While I understand that you will no longer be making regular contributions, please don't hesitate or feel awkward if one day you ever want to drop by to discuss something, make a bug fix, or even just pop in to say hi.

Thank you 🫡

[edreamleo]

@lieryan Many thanks for your kind an conciliatory words. I'll spend a day or two reconsidering whether I want to contribute to Rope and if so on what terms. You are under no obligation to accept those terms—they might well be the subject of negotiations.

[edreamleo]

I'll spend a day or two reconsidering whether I want to contribute to Rope and if so on what terms.

We must (eventually) agree on the following if I am to continue working on Rope.

I welcome extended conversation. I am open to changing my mind. There is no time limit.

Documentation

• docs 1: A Theory of Operation is worth having.

• docs 2: Docstrings for all inference-related classes, methods, and functions are worth having.

• docs 3: Exploring code with repl or breakpoint() is no substitute for documentation.

Code

• code 1: Rope's code (including its type hierarchy) is static (immutable).
  Few Python programs (including Rope) use Python's dynamic features.
  Thought experiment: Rope could be transliterated to C if a C version of Python's ast module existed.

• code 2: All classes defined in pyobjects.py and pyobjectsdef.py are in fact subclasses of PyObject.

• code 3: Single inheritance is preferable to multiple inheritance.

• code 4: Abstract classes are markers. These classes are not essential.

mypy

• mypy 1: Full mypy annotations are worth having.

• mypy 2: mypy's annotations assume (and require) immutable class hierarchies.

• mypy 3: The proper annotation for all instances of type-related classes is
  PyObject, not Union[PyObject, PyDefinedObject].

PRs

• PR 1: I must have a suitable code base for my work. This code base must:
  • Support full mypy annotations.
  • Clarify the Rope's existing type hierarchy.

Summary

We must resolve all the above assertions before I will do more work on Rope.

I am open to changing my mind. Some items are more negotiable than others.

I can't live without PR #664 or PR #677. I prefer #667, but PR #677 (or something like it) would work for me.

I promise to be satisfied with Rope's code if you accept one of these PRs.