Downgrading hard + version-bound dependencies

[atreyasha]

Hello @pbrisbin,

Thank you for making this awesome tool, it's really useful! Going back to issue #19, I think it might be really useful to have dependency management while downgrading a package. I understand you mentioned that you don't want to add it, but would it be possible to see how this works if I submit a PR?

If so, it would be great if you could advise me which area of the code you can envision this coming in; that would make it slightly quicker :)

[pbrisbin]

Hi there, as I mentioned in the original issue:

I'll gladly review any PRs that make this situation easier though

So do feel free to explore this.

it would be great if you could advise me which area of the code you can envision this

Hmm, I'm really not sure. It'll be a large change that will take some time to get done (I think) so an approach that keeps things mostly isolated from the existing code would provide the best chance of it getting reviewed and merged.

I imagine the process would have to work something like:

1. User is prompted with versions available for the original argument package
2. User makes her selection
3. Somehow, we find that version's PKGBUILD or SRCINFO or whatever
4. Insert dependency resolution function that uses this to (recursively) spit out more package-versions that need processing
5. The user's first argument, and all these new ones, are passed into the actual install-ignore function

(3) and (4) seems like the biggest risks, so I might begin by simply working on those in total isolation. If that can't be solved, there's no use in going any further.

As an alternative:

Downgrade's usage is flexible enough to be used as part of this process:

• It accepts full {package}-{version} arguments and will downgrade directly to that version without any prompting
• It accepts multiple arguments

Therefore, if some external tool could sort out the dependency graph of a multi-package downgrade, you could simply pass that tool's output to | xargs downgrade.

That approach is super attractive to me because it's very UNIX and leverages downgrade without bloating it. But the major downside is that downgrade is also good at taking a package name and prompting you based on Cache and ARM as to what you might want to install. That makes this tricky.

What about something like this:

% downgrade firefox --print | fzf | downgrade-dependencies | xargs downgrade

1. downgrade --print dumps the available {package}-{version}s in Cache or ARM
2. fzf now handles making your selection and printing that one out
3. downgrade-dependencies is a separate executable that does the heavy lifting and prints out more packages
4. Then xargs downgrade comes back to install and handle IgnorePkgs

Even if we do eventually decide to fold things into downgrade, working on an isolated downgrade-dependencies thing first is exactly how I'd approach it anyway.

[atreyasha]

Hi again. I agree with this and I think it is a good idea to develop a separate pipeline which essentially finds dependencies and uses downgrade to simply downgrade those packages, if an earlier version is still around in the archives. This also fits in with having things modular instead of combined/convoluted.

Do you have any recommendations for a specific language to use? I am not too familiar with perl so I would probably just go with shell, and perhaps some scripts in python for checking dependency graphs.

I will be quite busy in the coming 2-3 weeks, but will try to push a prototype in the new year. Will keep in touch.

PS: Sorry I closed the issue by mistake.

[pbrisbin]

Do you have any recommendations for a specific language to use?

Oh, that'd be totally up to you. Back when I wrote downgrade shell was really all I knew, so it was out of necessity. The choice probably comes down to how much of a distribution headache you want to have on your hands. Something interpret-able (shell, ruby, python) can just be shared and depend on the interpreter. Something compiled would either have to distributed as a binary (go, rust) or compiled during installation and so depend on that tool-chain (Haskell), which some find off-putting.

This logic feels complex enough that I'd want to do it in not shell, personally. Python, if you know it, seems like a good compromise between language features and ease of distribution.

[atreyasha]

Hi man!

So once again back to this issue. I have been thinking of how we could tackle this and I wanted to get your thoughts again.

So to make things simple, I would like to identify one example of a package that has a hard dependency, so that we both could refer to it. The simplest one I could think of is vim. So if we execute pacman -Si vim, we get the following output which I truncated for the important line:

Depends On      : vim-runtime=8.2.0510-1  gpm  acl  glibc  libgcrypt  pcre  zlib
                  libffi

So the hard dependency here is vim-runtime. Likewise, if we were to try downgrading vim only, we would come across an unresolved dependency error which says vim cannot be downgraded without downgrading vim-runtime. Something to the effect of:

loading packages...
resolving dependencies...
warning: cannot resolve "vim-runtime=8.1.1906-1", a dependency of "vim"

I have two perspectives here which I can't reconcile with on my own.

1. Downgrade should manage hard-dependencies

On one-hand, a user might expect downgrade to have its own dependency management like pacman and to manage hard-dependencies automatically. Thinking about the exact logic, this is in fact possible. To do it, we could replace sudo pacman -U with sudo pacman -Up to get the output of a prospective downgrade. If there is an unresolved dependency error, we catch the output and get the exact version to resolve this error. We can do this recursively till we resolve all dependencies. Once there are no printed errors, we simply execute sudo pacman -U with all the new pacakges as per what is already there in downgrade. In theory, this should work fine.

However, there are some possible issues. If you simply execute pacman -Si | grep "=", you will see examples of hard-dependencies of many packages. Only some of them have an exact hard dependency with =. Many have bounded ones, so I am not sure how we could handle this recursively.

Also, in the extremely unlikely case that there exist some circular hard dependencies, we might mess things up with forcing a downgrade. Hopefully pacman would be able to catch that and stop such a downgrade.

2. Downgrade is inherently interactive and should not automatically manage hard-dependencies

Another perspective here is that downgrade is generally an interactive tool which requires the user to look through possible downgrading options and select them. In that sense, the onus falls on the user to know the hard dependencies and downgrade with this in mind.

In the case of vim, the user should be aware of the vim-runtime hard dependency and should execute downgrade vim vim-runtime instead of downgrade vim, and should select the appropriate package versions.

In this perspective, downgrade not handling hard dependencies is a feature and not a bug.

Moving forward

I think the decision on what to do really depends on how we envision downgrade. If we go with perspective 1, we might need to think of how we can handle the bounded version dependencies. I think pacman should catch circular dependency issues, so really the only hindrance would be this version boundedness.

If we go with perspective 2, then we probably would not have to do anything further. This seems convenient, but somehow leaves a bad taste in my mouth knowing we did not try to do something that we could.

Would be great to get your thoughts.

[pbrisbin]

Thank you very much for all of these details!

I think, as you mentioned, the dependency problem is very hard and there are lots of unknowns regarding what cases we can robustly solve, which edge-cases we might have to choose not to solve, and how complex and maintainable will the solution be. And there's the question of if it's part of downgrade or a separate tool, if downgrade's UX should change to support dependencies being provided externally, if doing it in downgrade is worth a dependency on a "real" language or if it's feasible in shell...

I'm not looking for answers to all these questions today, I'm more painting a picture that it's a big ball of unknowns, which means the more we can de-risk it, and experiment in a non-disruptive way, the better.

To that end, here are my thoughts:

My gut says dependencies are not worth solving. So I lean towards (2). However, I do not want to quell your enthusiasm, so if I were going to explore (1), here's how I would get started:

1- I would fix the bug in (2)

😂 downgrade vim vim-runtime ought to work, but I don't think it does.

You should test this, but I think since we do pacman -U in a loop, even if the user passes both packages to be downgraded, it will fail on the first one. I'd consider that a bug worth fixing first, regardless of the other decisions.

If you fixed that bug, then downgrade vim-8.1.1906-1 vim-runtime-8.1.1906-1 will work too.

2- I would add a temporary, undocumented -p to downgrade

This -p/--print options tells downgrade to output the selected <package>-<version> on stderr after user selection (and do nothing else).

3- Tackle dependency resolution directly

Now you have this usage:

downgrade -p vim 2> >(downgrade-dependencies) | xargs downgrade

downgrade vim prompts for selection, user selects 8.1.1906-1, vim-8.1.1906-1 is printed.

downgrade-dependencies reads <package>-<version> on stdin, goes though all the complexity of dependency resolution, and outputs

vim-8.1.1906-1
vim-runtime-8.1.1906-1

on stdout, which xargs hands back to downgrade to install.

I know I'm like a broken record with this idea, but I think dependency resolution is going to take significant time, review, and iterations, and I'd hate to hold open a non-master branch on downgrade itself while this is all happening. I also don't want to put a lot of work into changing downgrade itself, when there's the possibility we discover something late that blocks us. Not to mention we can cover this hypothetical downgrade-dependencies in all sorts of unit tests trivially, which I think is important for this feature: we want to build something naive but correct, and then gradually optimize and refactor it, while ensuring it stays correct.

If we figure out that downgrade-dependencies can't or won't work, it's no big deal to abandon. If we like it, we'll know it's worth the next step of integrating it right into downgrade (or modifying how downgrade works to make the external integration easier for, or invisible to, users).

[atreyasha]

Thank you for the detailed feedback. While I am enthusiastic, I do feel that this would be too big a development now and there are perhaps some loose ends to tie up before we can do a full-fledged dependencies handling procedure for downgrade.

downgrade vim vim-runtime ought to work, but I don't think it does.

You should test this, but I think since we do pacman -U in a loop, even if the user passes both packages to be downgraded, it will fail on the first one. I'd consider that a bug worth fixing first, regardless of the other decisions.

Based on my personal tests and understanding, downgrade vim vim-runtime as well as downgrade vim-8.1.1906-1 vim-runtime-8.1.1906-1 work right now out-of-the-box. This is because of the pacman command for downgrading being sudo pacman -U "${pacman_args[@]}" "${to_install[@]}", which expands the to_install array as a space-separated sequence of packages. This would in fact execute sudo pacman -U just once with all of the pacman_args and packages in to_install, meaning that the dependency issues of vim and vim-runtime would be resolved in that instance. This works for me when I tested it on my notebook.

if we figure out that downgrade-dependencies can't or won't work, it's no big deal to abandon. If we like it, we'll know it's worth the next step of integrating it right into downgrade (or modifying how downgrade works to make the external integration easier for, or invisible to, users).

I agree with the workflow that you suggested and that would be an ideal way of going through the process without affecting the code in downgrade significantly as well. But yes this would be difficult to do. Based on this, I think perhaps we can leave this issue open as something to fix in the long(er)-run.

Short-term solution

As a pre-cursor to a long-term solution, I think it would be good if we could fix some version bounds handling directly in downgrade. What I mean by that is, perhaps we could slightly modify the code such that a user could input:

$ downgrade "vim>=8.1.1906"

And the selection output of this would be the filtered output with only relevant versions. This would contribute to two positive things: firstly allowing the user to directly filter version bounds and secondly allowing us to handle possible dependency bounds issues based on my comment yesterday.

Once this is solved, then we essentially would have all the tools necessary to handle dependencies and would just have to follow a procedure similar to what you mentioned to make it happen. What do you think?

[pbrisbin]

Based on my personal tests and understanding, downgrade vim vim-runtime as well as downgrade vim-8.1.1906-1 vim-runtime-8.1.1906-1 work right now out-of-the-box

Ah, of course! Sorry I was mistaken there.

What I mean by that is, perhaps we could slightly modify the code such that a user could input:

$ downgrade "vim>=8.1.1906"

And the selection output of this would be the filtered output with only relevant versions

That sounds like a great feature to me!

I would like to see the current non-interactive behavior retained with that feature too. I.e. if you pass in any argument (exact or bounds) that results in only a single choice, it is installed without interactive selection.

[pbrisbin]

Closing this in favor of #208.