Installation with straight

[ThibautVerron]

Hi,

I am trying to properly install and run eaf with straight, and it is hard to find documentation. I hope that this discussion can both help me and result in a usable guide for other users of straight.

So, first of all, the recipe I use (adapted from #321)

(use-package eaf
  :straight ( eaf
	      :type git
	      :host github
              :repo "manateelazycat/emacs-application-framework" ;; should it be emacs-eaf/... instead?
              :files ("*.el" "*.py" "core" "app"))
  :init (require 'eaf))

That is not enough, because it does not install the applications. One also needs to run the install-eaf.py script. But where? Straight uses separate directories for the source files and the built packages (typically .emacs.d/straight/repos and .emacs.d/straight/build).

If I run install-eaf.py in <repos>/eaf/ I get an app directory with all the applications in that folder. However, this does not get added to the load path, because emacs loads eaf.elc for the library and not eaf.el, and that file is in <builds>/eaf/. (require 'eaf) then looks for an app folder in <builds>/eaf/ and does not find any.

If I run install-eaf.py in <builds>/eaf/, the directory app is not created (or created somewhere else).

(I also have a folder <repos>/emacs-application-framework which appeared at some point, but for some reason straight does not build it. That might have to do with the difference in name between the recipe eaf and the repo.)

What seemed to work is to:

1. Run ./install-eaf.py in the <repos>/eaf/ folder
2. Copy the app folder over: cp -r <repos>/eaf/app <builds>/eaf/
3. Restart emacs

Is that the proper way of doing it?

[MatthewZMD]

The code you copied is out-of-date, namely

          :repo "manateelazycat/emacs-application-framework" ;; should it be emacs-eaf/... instead?
          :files ("*.el" "*.py" "core" "app"))

• Change it to emacs-eaf/ instead, it is the new git url.
• I think you also need to add "extension", *.json to :files, although I have no idea what :files mean in the straight context.

One also needs to run the install-eaf.py script. But where? Straight uses separate directories for the source files and the built packages (typically .emacs.d/straight/repos and .emacs.d/straight/build).

The behaviour of install-eaf.py is to add EAF app git repos to the directory of install-eaf.py, which normally should be the emacs-application-framework directory. If you run the interactive wrapper elisp function M-x eaf-install that calls install-eaf.py, it should run it in the (file-name-directory (locate-library "eaf")) directory.

EAF also uses its own eaf-add-subdirs-to-load-path elisp function to add the apps to the load-path, not depending on any package manager, as long as you have the eaf core directory in the load-path and (require 'eaf).

This works nicely if EAF is installed following the README instructions.

Is that the proper way of doing it?

I'm glad to see at least a workaround exists, I hate to see the situation that a package is unusable because of the package manager. It definitely far from the proper way, every time you update EAF and run install-eaf.py to update the installed applications, you'd have to copy it again to <builds>/eaf/, this is not good.

Unfortunately I don't use straight so I am not confident in what's the best way to do this, I can only guess at the moment.

If I run install-eaf.py in /eaf/, the directory app is not created (or created somewhere else).

So is there a install-eaf.py file in <builds>/eaf/? Does (file-name-directory (locate-library "eaf")) also evaluate to this directory? If you run it, does the script succeed?

If it succeeds, and you (require 'eaf), what's your load-path?

[ThibautVerron]

Change it to emacs-eaf/ instead, it is the new git url.

Okay, thanks.

I think you also need to add "extension", *.json to :files, although I have no idea what :files mean in the straight context.

I think that it specifies which files are transferred when building the package. The content of the folder in repos (incidentally, the correct one there is emacs-application-framework) is all the files, but the content of the build folder is only those which are listed in :files (and, in the case of .el, compiled in the process).

I suspect that the recipe I copied is also outdated in that sense, and that back then the app folder was part of the repo. I'm not quite sure of why I had to do the copy manually then, straight should have done it when restarting emacs. It could have to do with its caching mechanism.

But I now realize that streamlining the process might be as easy as adding :pre-build ("./install-eaf.py") to the straight recipe.

It could also be possible to take care both of the building and the additional requires by making a recipe pattern for the eaf applications, so that (straight-use-package 'eaf-pdf-viewer ...) would build the pdf-viewer and require the file. Whether it's a good idea could depend on how much slower it is to run the script 10 times for 1 application vs 1 time for 10 applications.

EAF also uses its own eaf-add-subdirs-to-load-path elisp function to add the apps to the load-path, not depending on any package manager, as long as you have the eaf core directory in the load-path and (require 'eaf).

Yes I tried to use that, but the problem is that the result of locate-directory is the folder with the elc, and not the one where the app file was created.

I'm glad to see at least a workaround exists, I hate to see the situation that a package is unusable because of the package manager.

Getting it to work with straight should be easier than with most package managers, because it's essentially trying to take care of the cloning and installation for us. It is easiest for pure elisp packages, but it is also able to install org, vterm or jupyter which also involve system calls. Making a recipe for eaf should be possible.

every time you update EAF and run install-eaf.py to update the installed applications, you'd have to copy it again to /eaf/, this is not good.

An easy (as long as we don't care about windows) fix for that is replacing the cp by ln -s.

So is there a install-eaf.py file in /eaf/? Does (file-name-directory (locate-library "eaf")) also evaluate to this directory? If you run it, does the script succeed?

Yes and yes and it seemed to (but no folder app appeared when I tried it).

I suspect that the fact that the json and extension files were not copied over is part of the why. Does the script try to cd where those files are? Or do parts of it silently fail if the files are not present?

If it succeeds, and you (require 'eaf), what's your load-path?

It was unchanged when I tried, which I attributed to the fact that the folder was not created (or created somewhere else).

I will try to get the recipe to work tomorrow.

[ThibautVerron]

Yes that's the repository. It looks like straight only symlinks the files by default, but the directories are copied. I guess that's why the script is executed in the other folder. Symlinking the ".el" files makes find-function work, but a case could be made for other files to be copied, I guess -- or everything to be symlinked.

Your answer to the other post should work for avoiding sudo, but if system files are missing, it will still fail. Querying for sudo at installation time should be possible, vterm and jupyter do it. It looks like they both do that from within the package files, and not as part of the straight recipe (this also lets them be installable via melpa).

(The syntax is the same because doom uses straight as its package manager.)

[MatthewZMD]

At this point, is there anything we can do to help you?

[MatthewZMD]

Your answer to the other post should work for avoiding sudo, but if system files are missing, it will still fail. Querying for sudo at installation time should be possible, vterm and jupyter do it. It looks like they both do that from within the package files, and not as part of the straight recipe (this also lets them be installable via melpa).

Do you know what those recipes look like?

[ThibautVerron]

My installation seems to work, thanks. But I'm still hoping we can reach a full recipe for installing eaf with straight/doom.

For now, the procedure would be to use the recipe, M-x eaf-install and restart emacs (so that straight copies over the app folder).

Do you know what those recipes look like?

I checked. For vterm the recipe doesn't do anything special, and the job is done in elisp the first time the package is loaded: https://github.com/akermu/emacs-libvterm/blob/f73644a07ce56712101d1568139a8f3c3b4a9e59/vterm.el#L134. For jupyter I can't find it, maybe I misremember the installation.

In any case, I don't see an easy way to make it work for eaf. Maybe I should ask straight's maintainer if there is a way to force a file to be copied instead of symlinked (for install-eaf.py), or to force the directories to be symlinked too (and then all that would be needed is an empty app in the repository).

[MatthewZMD]

For now, the procedure would be to use the recipe,

Could you share your current recipe, so at least someone else coming here can have a look?

restart emacs (so that straight copies over the app folder).

So right now straight does copy over the app folder it self, you don't need to do that manually anymore?

If I understand correctly, the only unfortunate outcome to your current installation is that, after you update EAF, you need to run M-x eaf-install-and-update manually again?

[ThibautVerron]

Could you share your current recipe, so at least someone else coming here can have a look?

Sure:

(use-package eaf
      :straight ( eaf
		  :type git
		  :host github
		  :repo "emacs-eaf/emacs-application-framework"
		  :files ("*")
		  ))

So right now straight does copy over the app folder it self, you don't need to do that manually anymore?

Upon restarting emacs, yes. Basically straight sees that the app folder was created, interprets it as a change in the sources, and "rebuilds" (in particular it compiles the new .el files).

In particular, running install-eaf.py in the builds directory is not the solution, because the elc would not be compiled.

If I understand correctly, the only unfortunate outcome to your current installation is that, after you update EAF, you need to run M-x eaf-install-and-update manually again?

And restart emacs.

Edit: Sorry for messing the conversation flow, I'm not familiar with this feature of github.

[ThibautVerron]

This recipe works out of the box (no M-x eaf-install and no restart):

(use-package eaf
      :straight ( eaf
		  :type git
		  :host github
		  :repo "emacs-eaf/emacs-application-framework"
		  :files ("*")
		  :pre-build (("python3" "install-eaf.py" "--install" "pdf-viewer" "--ignore-sys-deps"))
		  ))

But it requires specifying the list of apps in the command, and installing the dependencies manually.

Otoh the dependencies do not require a restart. Are they required at build time?

[dalanicolai]

So to comment on your request in syl20bnr/spacemacs#14992.

I have found that a problem here is that the eaf.el script adds the "wrong" load-path because the default directory points to the directory of the symlinked file. But if you replace that line in eaf-add-subdirs-to-load-path by:

(pending (list (file-name-directory (locate-library "eaf")))))

then it works fine.

Of course the load path gets added only when eaf.el gets loaded, so to load the packages from the app directory you should wrap them in a
with-eval-after-load, e.g.:

(with-eval-after-load 'eaf
  (require 'eaf-browser))

(And the 'autoload' app commands are not available before eaf gets loaded)

B.t.w. that latter recipe given in #321 look a little cleaner and works also.

(straight-use-package '(eaf :type git
                            :host github
                            :repo "manateelazycat/emacs-application-framework"
                            :files ("*.el" "*.py" "core" "app")))

[MatthewZMD]

Awesome! Thanks for the update, fixed!

[ThibautVerron]

I have found that a problem here is that the eaf.el script adds the "wrong" load-path because the default directory points to the directory of the symlinked file

I'm not sure to understand what is "right" and "wrong" in this context. The folder which should be added to the load path is the one with the "built" package, because that's where the elc files will be. Otoh the one which should be used for running the script is really the one with the source repository, because otherwise straight won't pick the new .el files for building.

In any case, both with that code and the old one, the directories added to the load path are the built ones, so this change doesn't seem to have introduced a bug. It's probably because locate-library points to the elc file, which is physically in the build directory.

I also think that it's a bit misleading to mark this as an answer (why the rush, anyway?). The recipe here is the exact same one I included in the first post, and it has all the problems already noted in this thread.

[MatthewZMD]

I'm not sure to understand what is "right" and "wrong" in this context. The folder which should be added to the load path is the one with the "built" package, because that's where the elc files will be. Otoh the one which should be used for running the script is really the one with the source repository, because otherwise straight won't pick the new .el files for building.

In any case, I don't see an easy way to make it work for eaf. Maybe I should ask straight's maintainer if there is a way to force a file to be copied instead of symlinked (for install-eaf.py), or to force the directories to be symlinked too (and then all that would be needed is an empty app in the repository).

@ThibautVerron I think forcing directories to be symlinked would be the reasonable thing to do here.

I also think that it's a bit misleading to mark this as an answer (why the rush, anyway?). The recipe here is the exact same one I included in the first post, and it has all the problems already noted in this thread.

I apologize for this.

[ThibautVerron]

The problem which I didn't realize when I wrote that is that if the app directory is symlinked, its files will never be byte- (and probably native-) compiled. The problem is the same with running the install script in the builds directory.

A solution could be to add byte compilation to the script (and running that script in the build directory, because a source directory can be shared between different versions of emacs)...

[MatthewZMD]

A solution could be to add byte compilation to the script (and running that script in the build directory, because a source directory can be shared between different versions of emacs)...

Do people always want to byte compile that we should add it to the install script is also a question.

[dalanicolai]

I think making use of the :pre-build ("./install-eaf.py") would be handy, but it would require to get the permissions to run the python script (I guess it just needs the executable permissions somehow, maybe like this or this?).

[MatthewZMD]

Done :)

[dalanicolai]

Okay, so unfortunately this does not help, because straight builds the package asynchronously and therefore it can neither show the prompts for installing the packages nor the prompt for the password (as far as I understand from here).

So now I think there are two options. Either the user simply restarts Emacs after running the install script (so that Emacs will copy the just created app directory to the build directory), or you create the prompts with elisp and subsequently pass that to the python script (e.g. for sudo see here).
But then you must split the python script in two parts, because one should not run pip commands with sudo (of course you could still run both scripts from a single elisp command).

Finally, the with-eval-after-load 'blocks' the install process when the app directory in the build directory is not yet available. So instead the user could use require and include the NO-ERROR flag. However, when using require all packages will load at startup, so I guess it would be nicer to run a use-package conditionally when the app directory within the build directory is available.

[dalanicolai]

And when you want to load the apps using use-package, I find that, at least for the browser, you should require 'eaf from those app .el files.

[ThibautVerron]

So, to summarize the problem. The steps to take are:

1. Download the source code of eaf
2. Copy/symlink the source code of eaf
3. Byte-compile the .el files of eaf
4. Install the dependencies (requires root)
5. Install the required apps
6. Copy/symlink the files from step 5 into the folder with eaf.elc (from step 2/3)
7. Byte-compile the .el files for the apps

Straight takes care of 1,2 and 3, as well as 6 and 7 if emacs is restarted after 4 and 5.
The installation script takes care of 4 and 5.

Now, getting the installation script to install the files in the build folder is not too difficult, all we have to do is run it like python install.py instead of ./install.py. And adding the byte-compilation step to the elisp wrapper around install.py is also not difficult. Then there would not be a need to restart emacs after running the script.

With those changes, how would you feel about running the installation script the first time the package is run after installing or updating? That would also make it possible to install the package with melpa.

[ThibautVerron]

Yes I have started writing a prototype, but it's going a bit slowly, sorry about that.

[MatthewZMD]

Good luck!

[ThibautVerron]

Ok, it took even longer than I thought and I'm still not done. Sorry about that.

Quick status update however: the code is on https://github.com/ThibautVerron/emacs-application-framework/tree/installation
Right now it almost works, but somehow the app folder doesn't get added to the load path and I'm not sure why.

I didn't really try it with Melpa, but I strongly suspect that it will not be able to correctly detect the Melpa version in that case.

[MatthewZMD]

Hi @ThibautVerron , thanks for your work! Could you please create a PR and label it as WIP, it would be easier for us to see your changes.

About melpa, I'm wondering what are you trying to do, because EAF isn't on melpa yet.

[ThibautVerron]

Okay I will do that.

About Melpa, that's definitely part of the problem, as it makes it difficult to test changes. As for what I'm trying to do, in principle with those changes it should be possible to install the package via melpa (similar to what jupyter and vterm do), so I wanted to include code to take care of that. But so far no luck.

[bevsxyz]

I got installation working with the following lines of use-package with straight.el

  (use-package eaf
    :straight (eaf
               :type git
               :host github
               :repo "emacs-eaf/emacs-application-framework"           
               :files ("*.el" "*.py" "core" "app" "*.json")
               :includes (eaf-pdf-viewer eaf-browser) ; Straight won't try to search for these packages when we make further use-package invocations for them
               :pre-build (("python" "install-eaf.py" "--install" "pdf-viewer" "browser" "--ignore-sys-deps"))
               )
    :init (evil-set-initial-state 'eaf-mode 'emacs)) ; Evil mode doesn't work well with eaf keybindings.

Load the apps separately with different use-package invocations.

 (use-package eaf-browser
    :custom
    (eaf-browser-continue-where-left-off t)
    (eaf-browser-enable-adblocker t))
(use-package eaf-pdf-viewer)

I am not sure about the :pre-build as I installed it in the terminal from the repo directory.

edit: :pre-build works very well now! Yay!!!! The code has been updated.

I deleted the emacs-application-framework directory from straight and restarted emacs to test. straight handles the cloning and building with grace.

[ezmiller]

When I run this, I get an error:

$ cd /Users/***/.emacs.d/straight/repos/emacs-application-framework/
$ python install-eaf.py --install pdf-viewer browser --ignore-sys-deps

[File error while Searching for program]