Working on autoimport.

[bagel897]

Hello!
I'm working on improving the autoimport for rope, so it can be integrated with autoimport and pylsp.
I want to change its data storage format to an sqllite3 database (saved to disk for persistence) and add the option to add all modules available in the path to the cache, and update as needed.
Couple of questions:

1. I would strongly prefer to use python 3.6+ features such as f-strings and not dealing with python2. Is maintaining python2/3.5 support a requirement for contributing?
2. As the index would have to be read/written to from memory, sqllite3 is the best option in the standard library + rope. If we can include 3rd party dependencies, sqlalchemy is a viable option. Is there a better/more preferred solution that is also performant?
3. Is there any specific guidelines I need to worry about such as about documentation?

[lieryan]

Hi @bageljrkhanofemus, thank you for your interest in improving rope and Python tooling in general.

Rope is in the process of deprecating Python 2, and it's likely Python 3.5 will also go away together with it when it does. Currently, I am planning the next release to drop Python 2 support.

The current rope import cache is basically just a pickled dictionary, with a very simple schema. I agree that they could be improved by moving towards a proper database. Sqlite would enable future possibilities to implement incremental updates (useful for larger codebases) or more sophisticated completion queries (e.g. fuzzy or fts completions).

In the past, I also have had thoughts on moving the import cache to sqlite. Currently, auto completion and auto import isn't an area of priority for me to implement the sqlite3 cache backend myself, but I'd be happy to provide any assistance you need and I'll accept code contributions towards this goal.

As the index would have to be read/written to from memory, sqllite3 is the best option in the standard library + rope. If we can include 3rd party dependencies, sqlalchemy is a viable option.

Currently rope doesn't have any dependencies on anything other than the standard library, and I prefer to keep it that way. There are many people that installs rope into the same site-packages/virtualenv as the one they use for their own project, rope not having additional dependencies minimises version conflicts with the user's project's dependencies. That said, I'm open to discussing adding dependencies if there is a sufficiently strong argument for it and there's minimal risks of conflicts with user projects.

Is there a better/more preferred solution that is also performant?

Currently, I think sqlite3 will be the best candidate for an improved rope caches. However, other alternatives I've considered in the past was dbm/shelves modules. Like sqlite3, these are also embedded databases that's available as part of the standard library, but unlike sqlite3, they are a key-value database instead of a relational database. Since the current cache was a pickled dictionary, my thinking was that moving to an embedded key-value database likely would more closely match the existing implementation and be an easier upgrade than to a relational database. That said, I don't think dbm would be a sufficiently worthwhile upgrade, since they would share many of the same issues as the current dictionary-based cache.

Is there any specific guidelines I need to worry about such as about documentation?

Nothing beyond the regular contribution guideline. Basically just update the CHANGELOG and if applicable, update docs/*.

The rope autoimport cache is an implementation detail. I don't expect any editor/IDE integrations to directly depend on specifics of the autoimport cache, so no need to worry about breaking compatibilities.

[bagel897]

Some follow up questions

1. Is it okay if I use pytest normally instead of unittest? I find the fixtures to be cleaner and better at avoiding testing errors
2. One major bottleneck in the program rn is the use of string based paths in Rope. Would it be possible to transition the paths to Pathlib and use its globs for much faster performance, either in the current pull request or a future one? This would allow me to massively speed up the autoimport code while maintaining ropes features such as moving/renaming files.

[lieryan]

Is it okay if I use pytest normally instead of unittest

I don't see why not. We currently only uses pytest for test discovery and reporting, if you're writing new tests I don't see any reason why we should not write pytest-style tests. However, please don't update existing test files just to convert them into pytest-style tests.

transition the paths to Pathlib

I'm not against using Pathlib, but switching to it for the whole project likely would not be as straightforward for a couple reasons:

1. The Resource/File/Folder classes are part of the Public API, so these constructs needs to remain as they are to avoid breaking existing editor integrations
2. For some editor integrations, rope works with file contents that are in the editor's unsaved buffer, in which case file/folder contents should not be read from the filesystem, but instead rope may queries the editor for the current content of the file. This should not be a problem when indexing site-packages which is usually not opened/modified by the editor and which rarely changes, but since rope has to index the user's project as well, the implementation would need to consider that. At least pylsp-rope deals with unsaved files, for example, though pylsp-rope currently does not implement autoimports.

That said if you're using pathlib just for internal implementations, that should be ok. I'm not entirely satisfied with how some of the resources classes are implemented either, as there are some parts that seems overly complicated.