Rename all package names with dashes to use underscore instead

[dagss]

Since we allow arbitrary Pytyhon expressions that refer to parameters like numpy.version in when-clauses and {{numpy.version}}, in packages that depend on NumPy, it seems to follow that we have to disable dashes in package names..

Working on a script to do this, but it has some problems (host-pkg-config turns into host-pkg_config...), pasting here as WIP for the record:

#!/usr/bin/env python
import os
import shutil
import subprocess

to_fix = []
for pkg in os.listdir('pkgs'):
    if '-' in pkg:
        to_fix.append(pkg.replace('.yaml', ''))

# Replace all occurences in YAML files without changing file location
if 0:
    for dirpath, dirnames, filenames in os.walk('.'):
        for filename in filenames:
            qpath = os.path.join(dirpath, filename)
            if not os.path.isfile(qpath):
                continue
            if '.git' in qpath:
                continue
            with open(qpath) as f:
                contents =  f.read()
            for x in to_fix:
                contents = contents.replace(x, x.replace('-', '_'))
            with open(qpath, 'w') as f:
                f.write(contents)

# Move files

for dirpath, dirnames, filenames in os.walk('.', topdown=False):
    for filename in filenames:
        qpath = os.path.join(dirpath, filename)
        target = qpath
        for x in to_fix:
            target = target.replace(x, x.replace('-', '_'))

        if qpath != target:
            head, tail = os.path.split(target)
            #os.makedirs(head)
            #subprocess.
            print '%s -> %s' % (qpath, target)

[dagss]

@cekees @certik @ahmadia how do you feel about banning dashes in package names? dashes are commonly used in package names in other systems, but they don't refer to packages in Python code..

One alternative would be to treat dash and underscore as same character, and then we'll have a mix everywhere. But it's more backwards compatible. I'm going to do this in Hashdist now as a temporary solution..

[cekees]

No feeling one way or the other. Supporting them doesn't seem like it's worth much time at this point, but if it's not hard might as well support common Unix file name conventions. Sent from my BlackBerry 10 smartphone on the Verizon Wireless 4G LTE network. From: Dag Sverre SeljebotnSent: Thursday, May 28, 2015 7:15 AMTo: hashdist/hashstackReply To: hashdist/hashstackCc: Chris KeesSubject: Re: [hashstack] Rename all package names with dashes to use underscore instead (#796)@cekees @certik @ahmadia how do you feel about banning dashes in package names? dashes are commonly used in package names in other systems, but they don't refer to packages in Python code..

One alternative would be to treat dash and underscore as same character, and then we'll have a mix everywhere. But it's more backwards compatible. I'm going to do this in Hashdist now as a temporary solution..

—Reply to this email directly or view it on GitHub.

[certik]

So a package name must be a Python identifier?

[dagss]

Yes, though I don't see why people would want to use unicode etc.

[certik]

What are the alternative options? Lots of packages have - in them and it's a pain when it doesn't work. I understand your motivation, but is there any other alternative?

[dagss]

Alternative 1:

Allow - in package names and it turns automatically to _ in expression contexts:

dependencies:
    build: [+python-yaml]

build_stages:
    - when python_yaml.version < 1.2: ...
      bash: |
          {{len(python_yaml.foo.bar) or 'baz'}}

That looks a bit ugly and inconsistent to me but it's an alternative.

Alternative 2:

Access some object as a dict:

build_stages:
    - when parameters['python-yaml'].version < 1.2: ...
      bash: |
          {{len(parameters['python-yaml'].foo.bar) - 2 or 1}}

I'm very against this myself.

Alternative 3:

Use another language for the expressions in YAML-files:

build_stages:
    - when python-yaml.version < 1.2: ...
      bash: |
          {{len(python-yaml.foo.bar) - 2 or 'baz'}}

This looks ugly to me (it's Python, but not..) and also is a bit tricky to implement, although not impossible at all (minimal viable is just replace '-' with '_' but leave ' - ' alone..)

A stronger counter-argument to me is that we use Python hook files, and then this syntax is not legal anyway. I do want expressions in hook files and in YAML files to be as similar as possible myself.

[dagss]

Hmm, something I actually like better is use python_dash_yaml in expressions to get the point across.

So:

build_stages:
    - when python_dash_yaml.version < 1.2: ...
      bash: |
          {{len(python_dash_yaml.foo.bar) - 2 or 'baz'}}

[certik]

How about this:

dependencies:
    build: [+python-yaml]

build_stages:
    - when "python-yaml".version < 1.2: ...
      bash: |
          {{len("python-yaml".foo.bar) or 'baz'}}

or this:

dependencies:
    build: [+python-yaml]

build_stages:
    - when P("python-yaml").version < 1.2: ...
      bash: |
          {{len(P("python-yaml").foo.bar) or 'baz'}}

Where P(package_name_str) returns you the package object for the given string.

[dagss]

Yes, it works. I think python_dash_yaml is prettier myself. And I think the prettiest option by far is just make python_yaml mandatory..

How about python_dash_yaml is supported but not recommended; it's just painful enough that people will feel encouraged to use python_yaml instead, but doesn't break anything.

[certik]

I think one should refer to packages by their names as strings, then there is no issue. This is what Spack does. How will this be implemented --- you have to create Python symbols for all packages and load it into some namespace and evaluate in that namespace? I think using something like P("...") is much cleaner.

[certik]

Also, you can't name a package that is also a Python keyword, and any future Python keyword... It seems that's a wrong way to approach it.

[dagss]

Dependencies are just other parameters. I feel the core of Hashdist depends on this -- remember Hashdist is much more like a programming language than Debian, RedHat etc.; you can parametrize a lot more. Then invariably you have to treat stuff the way you do in programming languages. So: A dependency is a parameter in a programming language (just like in Nix BTW)

So if you want to use P("foo") for anything, not just when there's a dash in there -- sorry, I can't accept writing

when P("platform") == "windows": ...

sources:
 - key:
     when P("version") == "1.2.3": tar.gz:asdfads

etc etc etc, that's a way too high price to pay for dashes in packages.

I guess I disagree with you about the Python keyword thingie. The big difference between us and Debian is simply that Hashdist is a programming language for generating stacks, Debian packages is not that at all.

Also I think the coupling with Python hooks will become stronger, not weaker, in time. Writing a new programming language from scratch, which is really what is needed, is just too much effort. (For instance, we should automatically compute pyver from python.version, that must be a Python hook. And so on). I guess I secretely believe YAML files might have been a mistake and would like to declare packages in Python as an alternative down the road. That's a different discussion :), but at least explains why I disagree with avoiding Python keywords being a bad idea.

Nix made its own functional language, and the other Nix derivative GUIX is based on a LISP dialect. We're based on Python, perhaps not the best choice, but much preferable to writing our own language from scratch like Nix did.

I guess I have a much stronger opinion about this now than earlier today :)

[dagss]

@certik what convention does Spack use? I guess all packages are Python identifiers by construction, but do you know how Spack would seperate yaml and python-yaml?

[certik]

Spack seems to just use strings. We can change yaml to something else, but that's a different discussion, where you never replied to my objections:

https://groups.google.com/d/msg/hashdist/DGg58Ed6vRI/SU_1BqKTPGsJ

[dagss]

Ah right, I though the class name had to be the same but there it is, package name cmake but class name Cmake (ugh..). I'll try to answer that email over the evening.

[dagss]

See email. If it helps my proposal, we could make it "packages must only be identifiers that are valid in both Python, JavaScript, Julia, Haskell, Ruby, and Go".

[certik]

How about making packages strings, but if the string also happens to be a valid Python identifier, you can refer to it as such, otherwise use P(...).

[dagss]

Yes, that works. I just prefer python_dash_yaml over P('python-yaml'), I think it's a lot cuter :) And will save me 20 minutes of implementation too (not really a consideration I guess). (What to do with packages named if is a bridge we cross when we get there IMO).

Perhaps @cekees can break the tie?

[certik]

I am fine with _dash_.

[cekees]

OK +1 dash. I'm always for saving 20 minutes.

On Thu, May 28, 2015 at 2:40 PM, Ondřej Čertík [email protected]
wrote:

I am fine with dash.

—
Reply to this email directly or view it on GitHub
#796 (comment).

[ahmadia]

Seems wonky to me. Maybe I'm not following the conversation.

On Thursday, May 28, 2015, Chris Kees [email protected] wrote:

OK +1 dash. I'm always for saving 20 minutes.

On Thu, May 28, 2015 at 2:40 PM, Ondřej Čertík <[email protected]
javascript:_e(%7B%7D,'cvml','[email protected]');>
wrote:

I am fine with dash.

—
Reply to this email directly or view it on GitHub
<#796 (comment)
.

—
Reply to this email directly or view it on GitHub
#796 (comment).

[dagss]

The only non-wonky alternative I see so far is not allowing python-yaml, only python_yaml. We can still have hit install do auto-correction if users type python-yaml..

But the _dash_ is a good migration path even if that is decided.

[dagss]

BTW this is not an original thought; Google App Engine can give you domain names with -dot- in them, e.g., myapp-dot-mymodule-dot-myversion.appspot.com.