Update "latest release" - sshfs for macFUSE #64

tormodwill · 2021-01-25T09:42:40Z

Hello

I have compiled sshfs for osxfuse ( now macFUSE )
I am now running SSHFS version 2.9 on macOS Mojave 10.14.6

Last login: Mon Jan 25 09:37:58 on console
Tormods-MacBook-Pro:~ tormod$ sshfs --version
SSHFS version 2.9
FUSE library version: 2.9.9
fuse: no mount point
Tormods-MacBook-Pro:~ tormod$

Why is it that version 2.5.0 is listed as the latest release ?
The SSHFS version 2.9 is working fine.

I did the following changes to the sshfs.c

force ssh-fs>sync_read = 0 in line 1827
( #57 )

changed # include <fuse_darwin.h> to #include <fuse.h>
( macfuse/macfuse#751 )

Would it be possible to have 2.9 as the latest release ?

The folder on https://github.com/osxfuse
Should be changed to https://github.com/macfuse

I also find that the text on the page
https://github.com/osxfuse/sshfs is misleading.
According to the page :
"The latest version and more information can be found on http://github.com/libfuse/sshfs"
Well, this is not correct if you use a mac. And if you don´t use a mac you would not be on the page in the first place.
The latest version for mac is 2.9, and the latest version can be found on the page :
https://github.com/osxfuse/sshfs

The latest version on the https://github.com/libfuse/sshfs will not work for mac.
( There is no FUSE 3 for OS X, FUSE library version: 2.9.9 for macFUSE macFUSE 4.0.4 )

Best regards

Tormod Willassen

Metacomet10 · 2021-01-28T01:37:54Z

Hi,

I was wondering what the proper procedure to compile and install v2.9 (as you have done) would be. I know how to use Terminal, but I'm very lost on how to take the files here and make them into a package that I can install. Any advice or pointers would be immensely helpful.

Thanks,
Metacomet10

tormodwill · 2021-01-28T11:25:55Z

How to compile and install SSHFS version 2.10.0 on macOS

The various steps described below will demand some knowledge in use of terminal.

For this tutorial I use the following versions of software:
Xcode_10.1 - for High Sierra 10.13.6
Command_Line_Tools_for_Xcode_10.1 - for High Sierra 10.13.6

Xcode_11.3.1 - for Mojave 10.14.6
Command_Line_Tools_for_Xcode_11.3.1 - For Mojave 10.14.6

If you run a newer version of macOS, you will find the correct version of Xcode here :
https://xcodereleases.com/

macFUSE - use latest release. Download from here :
https://github.com/osxfuse/osxfuse/releases
Use .dmg file ( already compiled )

glib-2.66.4
Download glib from here :
https://ftp.acc.umu.se/pub/GNOME/sources/glib/2.66/

SSHFS version 2.10.0
Download SSHFS from here :
https://github.com/libfuse/sshfs/releases/tag/sshfs-2.10
As of june 2021 this is the latest version that will run on mac.
The latest version on the https://github.com/libfuse/sshfs will not work for mac.
The latest version of libfuse shipping with macFUSE is 2.9.9. This means you are limited to SSHFS 2.x.
( There is no FUSE 3 for OS X, FUSE library version: 2.9.9 for macFUSE macFUSE 4.x.x )

Pyton 3.9.1
Download python from here :
https://www.python.org/downloads/release/python-391/

pkg-config 0.29.2
Download
https://pkgconfig.freedesktop.org/releases/?C=M;O=D
Description :
https://opensource.ncsa.illinois.edu/confluence/display/DESDM/Installing+pkg-config+from+source+for+OSX

Start the install / process

To get this working, you need glib.
in order to get glib working,
you need meson and ninja.
In order to start the process, you can have a look here:

https://mesonbuild.com/SimpleStart.html

Scroll down to the macOS part.

Install Xcode.
" Installing XCode is not sufficient by itself. You also need to start XCode' GUI application once. This will make XCode download and install more files that are needed for compilation."
This is a slow download. And takes a long time to install.

Install Command Line Tools for Xcode

Install Python 3
https://www.python.org/downloads/mac-osx/

Install meson and ninja.
pip3 install --user meson ninja

Your .bash_profile should have a PATH for meson and ninja :
Edit your .bash_profile using your favorite editor.
( I use vim )
$ vim .bash_profile

#Setting PATH for meson an ninja
export PATH=$PATH:/Users/YOUR_USERNAME_HERE/Library/Python/3.9/bin
#Setting PATH for Python 3.9
PATH="/Library/Frameworks/Python.framework/Versions/3.9/bin:${PATH}"
export PATH

When this is done, you can install glib.
This is probably the most difficult part.
Here is the instructions for installing glib :

% tar xf glib-@[email protected] # unpack the sources
% cd glib-@GLIB_VERSION@ # change to the toplevel directory
% meson _build # configure the build
% ninja -C _build # build GLib
[ Become root if necessary ]
% ninja -C _build install # install GLib

Install sshfs

Edit sshfs.c
Edit the sshfs.c before compiling: ( sshfs slow (0.2MB/sec) but sftp fast (6.0MB/sec) #57 )
Line 1724 in sshfs.c ( SSHFS version 2.10 )
change to sshfs.sync_read = 0
This will force sync read.
This change helped the speed on my line.

#if FUSE_VERSION >= 26
	/* Readahead should be done by kernel or sshfs but not both */
	if (conn->async_read)
		sshfs.sync_read = 0;
#endif

Edit sshfs.c
changed # include <fuse_darwin.h> to #include <fuse.h>
( SSHFS: Apple Silicon Native Support macfuse/macfuse#751 )
Edit test/meson.build
libfuse/sshfs@813b75b
--preserve=mode
Yes, works for linux. For BSD : no go.
The following parameter should be used for BSD / OSX : -fp .
Change the line beginning with command: to read:
command: ['cp', '-fp', '@input@', meson.current_build_dir() ])

( Compiling sshfs )
To build and install, use Meson (version 0.38 or newer) and Ninja. After extracting the sshfs tarball, create a (temporary) build directory and run Meson:

$ mkdir build; cd build
$ meson ..
To build, test and install SSHFS, you then use Ninja (running the tests requires the py.test Python module):
$ ninja
$ python3 -m pytest test/ # optional, but recommended
$ sudo ninja install

Best regards

Metacomet10 · 2021-02-03T21:12:57Z

Thank you for the in-depth walkthrough! I haven't been able to do this yet, but I'll let you know if I run into any problems.

luzik · 2021-02-09T07:14:26Z

There is also 2.10 release

tormodwill · 2021-02-09T21:09:32Z

What is the advantage by using 2.10 ? Why is 2.1 better than 2.5 or 2.9 ?

luzik · 2021-02-09T21:12:45Z

Please read release notes
https://github.com/libfuse/sshfs/releases?after=sshfs-3.3.0\

tormodwill · 2021-02-10T10:37:33Z

Thanks, luzik

So, it seems that version 2.10 is a newer ( more up to date ) version than version 2.5 or 2.9. Is this correct ?

I have been reading on this page :
https://github.com/osxfuse/sshfs/blob/master/ChangeLog

Reading the release notes was usefull.

I have one question :
In the release notes, i find the following for version SSHFS 3.6.0 :
"Improved performance under OS X"

So, I suppose this means that it is possible to run also the latest versions of SSHFS
on OS X / masOS.
How can this be done ?

My understanding was that the latest FUSE library for OS X / masOS is 2.9.9.
My understanding was that the SSHFS 3.x versions needed fuse library version 3 or later on OS X / masOS ?

Release notes from SSHFS 3.0.0:
"sshfs now requires libfuse 3.1.0 or newer"

How can I run FUSE library version 3 or later on OS X / masOS ?
Currently I run FUSE library version: 2.9.9. This version is in the "LATEST VERSION" of macFUSE. ( macFUSE 4.0.5 )

tormodwill · 2021-02-10T11:05:25Z

Ping, @Nikratio

bfleischer · 2021-02-10T12:18:33Z

The latest version of libfuse shipping with macFUSE is 2.9.9. This means you are limited to SSHFS 2.x.

Is there any important feature available in SSHFS 3.7.1, but not in version 2.5?

tormodwill · 2021-02-10T13:20:01Z

The advantages running a newer version :
SSHFS Version 3.6.0:
*Added --verbose option.
*Fixed a number of compiler warnings.
*Improved performance under OS X.

Some serious bugs have not been handled :
force ssh-fs>sync_read = 0 in line 1827
( #57 )
changed # include <fuse_darwin.h> to #include <fuse.h>
( macfuse/macfuse#751 )

I also find a list of various features here :
https://github.com/libfuse/sshfs/releases
If the features are important or not to the user, depends on the various user senario.
A feature that is urgent and important for me, might not be of interest for you.

You are also not answering my question :
"How can I run FUSE library version 3 or later on OS X / masOS ?"

According to the text under release notes this seems to be possible :
libfuse 3.9.3
Fixed compilation under OS X and µClibc.

bfleischer · 2021-02-10T14:37:42Z

The changes you listed seem to boil down to performance improvements. Backporting the relevant code changes to version 2.10 should be doable. Don't get me wrong, porting libfuse 3 to macOS is on my to do list, but other items on the list have a higher priority.

According to the text under release notes this seems to be possible :
libfuse 3.9.3
Fixed compilation under OS X and µClibc.

libfuse 3 might compile on macOS, but this does not mean that libfuse 3 will actually work. libfuse 3 currently lacks the required patches for supporting macOS. From what I can tell, the author of the comment you mentioned only added an #ifdef.

Currently the macFUSE kernel extension comes with (partial) support for FUSE ABI 7.18. This is not going to change from updating libfuse to version 3. In other words, most new libfuse 3 features will not be available on macOS. In case libfuse 3 requires newer ABI features, it will not work at all. What I'm trying to say is that updating libfuse makes most sense when updating the kernel extension in tandem.

You are also not answering my question :
"How can I run FUSE library version 3 or later on OS X / masOS ?"

You would have to add support for macOS to libfuse 3.

tormodwill · 2021-02-10T21:05:02Z

Thanks for the answer, Benjamin Fleischer.

The version 2.5 can not mount under /Volumes on my machines. When I run version 2.9, I am able to mount under /Volumes. This is the behavior I see. ( Maybe I have a strange config )

It is not obvious that you have to do manual changes to the sshfs.c file. If you don´t do these changes it will not compile. ( macfuse/macfuse#751 )

And also had to make manual adjustments to make a temporary fix for the preformance. ( #57 )

Making a .dmg file of the SSHFS version 2.10 will fix some issues for a lot of users.

When I started using the sshfs program on macOS, I wanted it to work as seamlessly as my pcloud drive. I wanted the ssh program to connect by itself at boot. I wanted to see the icon for the ssh drive in finder, and I wanted to be able to disconnect and connect using an icon on the desktop ( without any passwords ). To be able to do this a lot of fiddling and scripts has been needed. None of this has been working at first attempt.

I suppose the mac users are demanding. We are used to things that "just works".

Timmmm · 2021-04-08T10:43:38Z

How does SSHFS 3.7.0 improve performance under OS X if osxfuse/macFUSE/FUSE for Mac only supports SSHFS 2.x? Is there a way to run SSHFS 3.x on Mac?

The proliferation of old/incompatible versions and renamed projects makes this very confusing!

knaccc · 2021-10-24T16:20:31Z

@tormodwill Thank you!! This worked for me, and SSHFS performance increased for me from 20MB/s read to 115MB/s read over a 1GbE LAN.

I have brew and Xcode installed, so the instructions were simpler for me:

brew install python3 meson ninja glib wget
wget https://ftp.acc.umu.se/pub/GNOME/sources/glib/2.66/glib-2.66.8.tar.xz
wget https://github.com/libfuse/sshfs/releases/download/sshfs-2.10/sshfs-2.10.tar.gz
wget https://pkgconfig.freedesktop.org/releases/pkg-config-0.29.2.tar.gz
tar xf glib-2.66.8.tar.xz
tar xf sshfs-2.10.tar.gz
tar xf pkg-config-0.29.2.tar.gz

edit sshfs-2.10/sshfs.c and replace:
# include <fuse_darwin.h>
with
# include <fuse.h>
and on line 1724 change
sshfs.sync_read = 1;
to
sshfs.sync_read = 0;

edit sshfs-2.10/test/meson.build
and change the line beginning with command: to read: command: ['cp', '-fPp',

cd sshfs-2.10
mkdir build; cd build
meson ..
ninja

And voila, a brand new sshfs is built.

knaccc · 2021-10-24T17:20:59Z

One thing - on MacOS, for some reason when I read from an sshfs mount connected to a local LAN server via 1GbE, it can be as slow as 17-25 MB/s for the first 30 seconds, then rises to 110MB/s for the remainder of the read. This happens consistently.

My options are big_writes,iosize=1048576,reconnect,ServerAliveInterval=15,ServerAliveCountMax=3,allow_other,noapplexattr,noappledouble,defer_permissions

Does anyone have any suggestions please?

ricfregian · 2022-03-25T07:11:36Z

My machine is a macOS BigSur 11.6 and I followed your steps @tormodwill to install sshfs 2.10. However in the pytest step I get this:

`giancaceresvera@~/Downloads/sshfs-2.10/build$ python3 -m pytest test/
========================================================================= test session starts ==========================================================================
platform darwin -- Python 3.9.10, pytest-7.1.1, pluggy-1.0.0 -- /usr/local/opt/[email protected]/bin/python3.9
cachedir: .pytest_cache
rootdir: /Users/giancaceresvera/Downloads/sshfs-2.10/build/test, configfile: pytest.ini
collected 4 items

test/test_sshfs.py::test_sshfs[0-False] SKIPPED (Can't find fusermount executable) [ 25%]
test/test_sshfs.py::test_sshfs[0-True] SKIPPED (Can't find fusermount executable) [ 50%]
test/test_sshfs.py::test_sshfs[1-False] SKIPPED (Can't find fusermount executable) [ 75%]
test/test_sshfs.py::test_sshfs[1-True] SKIPPED (Can't find fusermount executable) [100%]

=========================================================================== warnings summary ===========================================================================
test/conftest.py:66
/Users/giancaceresvera/Downloads/sshfs-2.10/build/test/conftest.py:66: PytestDeprecationWarning: @pytest.yield_fixture is deprecated.
Use @pytest.fixture instead; they are the same.
@pytest.yield_fixture(autouse=True)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
======================================================================= short test summary info ========================================================================
SKIPPED [4] test/test_sshfs.py:32: Can't find fusermount executable
==================================================================== 4 skipped, 1 warning in 0.01s =====================================================================
`
I suppose the fusermount has to be found but I'm not sure. Any suggestions please ?

tormodwill · 2022-05-24T18:14:07Z

@tormodwill Thank you!! This worked for me, and SSHFS performance increased for me from 20MB/s read to 115MB/s read over a 1GbE LAN.

I have brew and Xcode installed, so the instructions were simpler for me:
brew install python3 meson ninja glib wget
wget https://ftp.acc.umu.se/pub/GNOME/sources/glib/2.66/glib-2.66.8.tar.xz
wget https://github.com/libfuse/sshfs/releases/download/sshfs-2.10/sshfs-2.10.tar.gz
wget https://pkgconfig.freedesktop.org/releases/pkg-config-0.29.2.tar.gz
tar xf glib-2.66.8.tar.xz
tar xf sshfs-2.10.tar.gz
tar xf pkg-config-0.29.2.tar.gz
edit sshfs-2.10/sshfs.c and replace: # include <fuse_darwin.h> with # include <fuse.h> and on line 1724 change sshfs.sync_read = 1; to sshfs.sync_read = 0;

edit sshfs-2.10/test/meson.build and change the line beginning with command: to read: command: ['cp', '-fPp',
cd sshfs-2.10
mkdir build; cd build
meson ..
ninja
And voila, a brand new sshfs is built.

The correct edit of test/meson.build
change the line beginning with command: to read:
command: ['cp', '-fp', '@input@', meson.current_build_dir() ])

knaccc · 2022-05-25T00:00:51Z

@tormodwill thanks - what is the consequence of changing from '-fPp' to '-fp'?

tormodwill · 2022-05-25T09:33:10Z

This is my Interpretation

@tormodwill thanks - what is the consequence of changing from '-fPp' to '-fp'?

This is my interpretation : OSX is based on BSD.
https://www.freebsd.org/cgi/man.cgi?cp
-P is not needed, and I was not able to compile with this flag

-P No symbolic links are followed. This is the default if the -R option is specified.

-p Cause cp to preserve the following attributes of each source file
in the copy: modification time, access time, file flags, file mode,
ACL, user ID, and group ID, as allowed by permissions.

   If the user ID and group ID cannot be preserved, no error message
   is displayed	and the	exit value is not altered.

   If the source file has its set-user-ID bit on and the user ID can-
   not be preserved, the set-user-ID bit is not	preserved in the
   copy's permissions.	If the source file has its set-group-ID	bit on
   and the group ID cannot be preserved, the set-group-ID bit is not
   preserved in	the copy's permissions.	 If the	source file has	both
   its set-user-ID and set-group-ID bits on, and either	the user ID or
   group ID cannot be preserved, neither the set-user-ID nor set-
   group-ID bits are preserved in the copy's permissions.

https://man7.org/linux/man-pages/man1/cp.1.html
-P, --no-dereference
never follow symbolic links in SOURCE

-p same as --preserve=mode,ownership,timestamps

tormodwill · 2022-05-25T09:41:46Z

https://github.com/tormodwill/macSSHFS

Timmmm mentioned this issue Apr 8, 2021

Clarify naming & versions macfuse/macfuse#796

Closed

knaccc mentioned this issue Oct 24, 2021

Performance flags for use over sshfs (Mac) rfjakob/gocryptfs#614

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update "latest release" - sshfs for macFUSE #64

Update "latest release" - sshfs for macFUSE #64

tormodwill commented Jan 25, 2021

Metacomet10 commented Jan 28, 2021

tormodwill commented Jan 28, 2021 •

edited

Loading

Metacomet10 commented Feb 3, 2021

luzik commented Feb 9, 2021

tormodwill commented Feb 9, 2021

luzik commented Feb 9, 2021

tormodwill commented Feb 10, 2021 •

edited

Loading

tormodwill commented Feb 10, 2021

bfleischer commented Feb 10, 2021 •

edited

Loading

tormodwill commented Feb 10, 2021

bfleischer commented Feb 10, 2021

tormodwill commented Feb 10, 2021

Timmmm commented Apr 8, 2021

knaccc commented Oct 24, 2021 •

edited

Loading

knaccc commented Oct 24, 2021 •

edited

Loading

ricfregian commented Mar 25, 2022

tormodwill commented May 24, 2022

knaccc commented May 25, 2022

tormodwill commented May 25, 2022 •

edited

Loading

tormodwill commented May 25, 2022

Update "latest release" - sshfs for macFUSE #64

Update "latest release" - sshfs for macFUSE #64

Comments

tormodwill commented Jan 25, 2021

Metacomet10 commented Jan 28, 2021

tormodwill commented Jan 28, 2021 • edited Loading

How to compile and install SSHFS version 2.10.0 on macOS

Start the install / process

Metacomet10 commented Feb 3, 2021

luzik commented Feb 9, 2021

tormodwill commented Feb 9, 2021

luzik commented Feb 9, 2021

tormodwill commented Feb 10, 2021 • edited Loading

tormodwill commented Feb 10, 2021

bfleischer commented Feb 10, 2021 • edited Loading

tormodwill commented Feb 10, 2021

bfleischer commented Feb 10, 2021

tormodwill commented Feb 10, 2021

Timmmm commented Apr 8, 2021

knaccc commented Oct 24, 2021 • edited Loading

knaccc commented Oct 24, 2021 • edited Loading

ricfregian commented Mar 25, 2022

tormodwill commented May 24, 2022

knaccc commented May 25, 2022

tormodwill commented May 25, 2022 • edited Loading

tormodwill commented May 25, 2022

tormodwill commented Jan 28, 2021 •

edited

Loading

tormodwill commented Feb 10, 2021 •

edited

Loading

bfleischer commented Feb 10, 2021 •

edited

Loading

knaccc commented Oct 24, 2021 •

edited

Loading

knaccc commented Oct 24, 2021 •

edited

Loading

tormodwill commented May 25, 2022 •

edited

Loading