Consider using (and extending) dephell

[ceball]

The extending would be for conda, I imagine.

[orsinium]

DepHell has some kind of support for conda: search, resolve, lock, convert. Not installation, though.

[ceball]

Cool. Not sure how you found this hidden away project, but thanks for the info! I really want to try out dephell but never have time.

Does dephell translate package names at all when moving between ecosystems (or allow a way for someone to map them)? We've encountered various complications between pypi and anaconda.org, including e.g.

• If I do conda install msgpack, my python gets downgraded from 3.6 to 2.7 and I get an old msgpack. Turns out I should have done conda install msgpack-python. Meanwhile, https://pypi.org/project/msgpack-python/ says "This package is deprecated. Install msgpack instead."

• If I do pip install graphviz, I'll be able to import graphviz, but I won't be able to render without also installing the underlying non-python graphviz package. E.g. apt-get install graphviz or conda install graphviz. If I'd been using apt or conda, then I could instead have just installed python-graphviz (and graphviz refers to the underlying, non-python package).

• If I do pip install dask I get dask plus the minimum dependencies it requires. If I do conda install dask I get all the dependencies.

[orsinium]

To be honest, I'm not a conda user. I dived in it only once, when I was implementing its support for DepHell. I'm not sure that I handle names mismatching in some way. We have names mapping for pypi name <-> import name but not for conda. However, I definitely have seen environment.yml generation from poetry's pyproject.toml with dephell in a wild. So, I assume, it works for simple cases.

Anyway, you can just try it, it's free :) I see a lot of packages in dephell package search --repo=conda msgpack output. package show also finds something:

❯ dephell package show --repo=conda msgpack 
{
  "authors": [],
  "description": "",
  "latest": "0.2.3",
  "license": null,
  "links": {},
  "name": "msgpack",
  "updated": "1970-01-01"
}

[orsinium]

What features do you miss in DepHell? Making your own dependency management tool is a big step and a lot of work and suffering. Maybe, we can, as you said in the title, just extend dephell instead. Try it, give it a go, it's much easier.

[ceball]

I unfortunately have no idea what features dephell is missing yet - I have to investigate to find out :) That part is unfortunately not free in that it takes time. But here is a quick demo of the specific problems I mentioned above, in case it helps...

Get sampleproject and add graphviz and msgpack as dependencies

$ git clone https://github.com/pypa/sampleproject.git && cd sampleproject
$ git diff
diff --git a/setup.py b/setup.py
index a9960a8..0ddf234 100644
--- a/setup.py
+++ b/setup.py
@@ -150,7 +150,7 @@ setup(
     #
     # For an analysis of "install_requires" vs pip's requirements files see:
     # https://packaging.python.org/en/latest/requirements.html
-    install_requires=['peppercorn'],  # Optional
+    install_requires=['graphviz','msgpack'],  # Optional
 
     # List additional groups of dependencies here (e.g. development
     # dependencies). Users will be able to install these using the "extras"
@@ -161,7 +161,6 @@ setup(
     # Similar to `install_requires` above, these must be valid existing
     # projects.
     extras_require={  # Optional
-        'dev': ['check-manifest'],
         'test': ['coverage'],
     },

Use dephell to convert from setup.py to conda:

$ dephell deps convert --from-format=setuppy --from-path=setup.py --to-format=conda --to-path=test.yaml
WARNING cannot find config file 
INFO converted 
$ cat test.yaml 
name: sampleproject
channels:
- defaults
dependencies:
- coverage
- graphviz
- msgpack

msgpack is not the right name for conda:

$ conda env create -f test.yaml 
Collecting package metadata (repodata.json): done
Solving environment: failed

ResolvePackageNotFound: 
  - msgpack

(If I want msgpack in python with conda, I do conda install msgpack-python.)

With conda, graphviz is not the python package name:

$ conda install graphviz
[...]
$ python
[...]
>>> import graphviz
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'graphviz'

(If I want graphviz in python with conda, I do conda install python-graphviz. Yes, the naming is the other way round from the msgpack example above - I have no idea why there's no namespace convention even, but I have to deal with it anyway.)

You might say that conda should not be so different from the python ecosystem, i.e. the fault lies with conda. I agree there are namespace issues that don't appear to have been thought through (I might be wrong), but on the other hand note that e.g. pip install graphviz does not actually produce a working installation of graphviz:

$ pip install graphviz
[...]
$ python
[...]
>>> from graphviz import Digraph
>>> dot = Digraph()
>>> dot.render('test123', view=True)
Traceback (most recent call last):
  File "/[...]/lib/python3.7/site-packages/graphviz/backend.py", line 159, in run
    proc = subprocess.Popen(cmd, startupinfo=get_startupinfo(), **kwargs)
  File "/[...]/lib/python3.7/subprocess.py", line 800, in __init__
    restore_signals, start_new_session)
  File "/[...]/lib/python3.7/subprocess.py", line 1551, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'dot': 'dot'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/[...]/lib/python3.7/site-packages/graphviz/files.py", line 209, in render
    quiet=quiet)
  File "/[...]/lib/python3.7/site-packages/graphviz/backend.py", line 206, in render
    run(cmd, capture_output=True, cwd=cwd, check=True, quiet=quiet)
  File "/[...]/lib/python3.7/site-packages/graphviz/backend.py", line 162, in run
    raise ExecutableNotFound(cmd)
graphviz.backend.ExecutableNotFound: failed to execute ['dot', '-Tpdf', '-O', 'test123'], make sure the Graphviz executables are on your systems' PATH```

I would need to install graphviz via apt or similar. (Or there would need to be a lot more things in the binary wheels people produce for projects I depend on...)

[ceball]

Making your own dependency management tool is a big step and a lot of work and suffering. Maybe, we can, as you said in the title, just extend dephell instead. Try it, give it a go, it's much easier.

Believe me, I'm not going to write my own dependency management tool :) pyctdev mainly just uses other tools, and doesn't do anything itself...except for providing basic support for translating between anaconda and pypi package names, because it's something we need. And I also wanted to add support for package name to import name(s) mapping to pyctdev, for use in testing packages. All this package name mapping is something I was considering breaking out into its own project (which pyctdev would then use) - and this is how I found dephell, when doing a new background search for similar projects. I was very happy to see a project to do some of this had been created since last time I searched! (Unfortunately, dephell appeared well after pyctdev (and its predecessor) was already "finished" and in use, otherwise I might be using it already...)

[jbednar]

I would need to install graphviz via apt or similar. (Or there would need to be a lot more things in the binary wheels people produce for projects I depend on...)

Which is of course why one would want to use Conda for such things...

[ceball]

Which is of course why one would want to use Conda for such things...

Right. And not just as a user - if you are the author of a python library/app, and you depend on e.g. graphviz, you can release a conda package depending on python-graphviz and know it will work with a high chance of success, on linux, windows, and mac (all without any special privileges) That takes away a lot of support burden. (Except for all the people coming to you asking for help doing it without conda! ;) )