run cell2location on M1 silicon chip BookPro #133

lefterov · 2022-03-20T17:52:14Z

lefterov
Mar 20, 2022

Has anyone been able to install and run cell2locations tutorials on Apple laptops with an M1 chip? it seems that the conflict with jaxlib built using AVX instructions remains unresolved... it is a problem no matter the version of python, Theano, or pymc3.

Any idea how to work around it, or do we have to wait until all kinds of M1-related issues are solved...?

vitkl · 2022-03-20T19:18:40Z

vitkl
Mar 20, 2022
Maintainer

Hi Very good question. I would think that it is impractical to analyse Visium data on most M1 laptops (unless you have more than 16GB RAM) - but maybe 1-2 sections can be analysed. I would say that you have to wait until we delete pymc3 and theano from cell2location repo (it is currently needed exclusively for cell2location-WTA model for Nanostring data). Pytorch seems to work on M1 Mac.

…

On Sun, 20 Mar 2022, 17:52 Iliya Lefterov, ***@***.***> wrote: Has anyone been able to install and run cell2locations tutorials on Apple laptops with an M1 chip? it seems that the conflict with jaxlib built using AVX instructions remains unresolved... it is a problem no matter the version of python, Theano, or pymc3. Any idea how to work around it, or do we have to wait until all kinds of M1-related issues are solved...? — Reply to this email directly, view it on GitHub <#133>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFMFTVZNU7RJ6UQLUXAFVRLVA5QVTANCNFSM5RFZ5CRQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

6 replies

lefterov Mar 22, 2022
Author

Apparently, it is not just the M1 (at least in my hands...). here is what happens on Intel Xeon W chip/MacOS Monterey:

while installing cell2location from the latest git repository in a python3.10 venv I get incompatible packages:
the installed arviz is 0.11.4, the required is 0.10.0
the installed scvi-tools is 0.15.2, the required is 0.14.6
I didn't find a way to solve the problem...

Any idea?

vitkl Mar 23, 2022
Maintainer

Re M1: Nice Mac you have :) You can create a branch of cell2location and delete all pymc3 code yourself as well.
Do you know if pytorch already supports the use of M1 GPU (not just CPU as it did initially)?

Re installing on macs - I have the same issues, also issues with installing anndata on Mac. The following works for me:

export PYTHONNOUSERSITE="aaaaa"
conda create -y -n test_c2l python=3.8
conda activate test_c2l
pip install pyro-ppl
conda install -y -c anaconda hdf5 pytables netcdf4
pip install git+https://github.com/BayraktarLab/cell2location.git#egg=cell2location[tutorials]
conda activate test_c2l
python -m ipykernel install --user --name=test_c2l --display-name='Environment (test_c2l)'

lefterov Apr 1, 2022
Author

@vitkl: on M1 Max while running cell2location_short_demo notebook, at the point to define r:
r = cell2location.run_cell2location(
I could not avoid the warning;

When trying Mapping human lymph node cell types to 10X Visium,

the Error message is that torch has not been activated properly.

in the script that you are suggesting above I used directly
pip install cell2location

in the new environment anndata is insltalled:
(pytorch_env) MacBook-M1-16:ST8059048:% conda list anndata <master ✗>

packages in environment at /Users/iliya/opt/anaconda3/envs/pytorch_env:

Name Version Build Channel

anndata 0.8.0 pypi_0 pypi

and I guess all required "torches" are too:

Name Version Build Channel

pytorch-lightning 1.5.10 pypi_0 pypi
torch 1.11.0 pypi_0 pypi
torchmetrics 0.7.3 pypi_0 pypi

now: should I use this environment to 1/3 tutorial?

Thanks and have a nice weekend

vitkl Apr 2, 2022
Maintainer

I see. I have not actually run the workflow on my laptop - I only used the created environments for code development in pycharm.

I guess I would prefer to have a combination of simpler M1 laptops with A100 80GB GPU server to having only the best M1 laptops :)

vitkl Apr 2, 2022
Maintainer

Sorry, it is hard to help with this issue. If you find any solution would be great if you contribute installation instructions.

lefterov · 2022-03-24T02:35:48Z

lefterov
Mar 24, 2022
Author

this is the type of M1s my postdocs work on... 🙂 from time to time I have to jump in and provide some help... yes, it is possible to install pytorch and it should work on M1 (I believe)... https://towardsdatascience.com/yes-you-can-run-pytorch-natively-on-m1-macbooks-and-heres-how-35d2eaa07a83 [https://miro.medium.com/max/1200/1*hsIzbaZLW6bsrwhvYkYWzQ.jpeg]<https://towardsdatascience.com/yes-you-can-run-pytorch-natively-on-m1-macbooks-and-heres-how-35d2eaa07a83> Yes, You Can Run PyTorch Natively on M1 MacBooks, and Here’s How | by Dario Radečić | Towards Data Science<https://towardsdatascience.com/yes-you-can-run-pytorch-natively-on-m1-macbooks-and-heres-how-35d2eaa07a83> M1 Macbooks aren’t that new anymore. Somehow, installing Python’s deep learning libraries still isn’t a straightforward process. At least with TensorFlow. PyTorch is different. Today you’ll learn how… towardsdatascience.com  on the more practical side, (following one of your suggestions in Discussions), on 16" M1 Max, I got to the point when "import cell2location" is possible (for me) on the command line of python 3.9.10 but only if the installation is within a python virtual env. the same strategy works much faster and elegantly on a CentOs 8 Stream installed on a Dell with Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz/8 core; python there is 3.10.2 (I noticed that Theano and pymc3 are no longer problems with 3.10) However: * on the M1 the interpreter does not pick up mkl and mkl-service even though they are there; removing and reinstalling the packages with conda or pip does not help; people say that the waning, and that is just a warning, can be ignored; * * on CentOS, the problem is similar, although a little different: Spyder displays a warning, that Rtree is missing, even though it is there; again it is just a warning. I have problems running the latest version of PyCharm but it is possible to hit the same problem there as well... * * I don't know what will happen if I try Jupyter... * * there is no problem installing anndata on M1. So..., we don't give up... even though the graduate students are moving faster analyzing our spatial transcriptomic datasets with Seurat... 🙁 thanks, again! and will keep you posted. will appreciate it if you don't post this in the Discussions Best,

…

-I. -- Iliya Lefterov, MD, PhD Professor, University of Pittsburgh ***@***.*** ***@***.*** 412 383 6906 <http://voice.google.com/calls?a=nc,%2B14123836906> - office 412 576 9923 <http://voice.google.com/calls?a=nc,%2B14125769923> - mobile 412 449 9195 <http://voice.google.com/calls?a=nc,%2B14124499195> - mobile

________________________________ From: Vitalii Kleshchevnikov ***@***.***> Sent: Wednesday, March 23, 2022 7:20 PM To: BayraktarLab/cell2location ***@***.***> Cc: Lefterov, Iliya ***@***.***>; Author ***@***.***> Subject: Re: [BayraktarLab/cell2location] run cell2location on M1 silicon chip BookPro (Discussion #133) Re M1: Nice Mac you have :) You can create a branch of cell2location and delete all pymc3 code yourself as well. Do you know if pytorch already supports the use of M1 GPU (not just CPU as it did initially)? Re installing on macs - I have the same issues, also issues with installing anndata on Mac. The following works for me: export PYTHONNOUSERSITE="aaaaa" conda create -y -n test_c2l python=3.8 conda activate test_c2l pip install pyro-ppl conda install -y -c anaconda hdf5 pytables netcdf4 pip install git+https://github.com/BayraktarLab/cell2location.git#egg=cell2location[tutorials] conda activate test_c2l python -m ipykernel install --user --name=test_c2l --display-name='Environment (test_c2l)' — Reply to this email directly, view it on GitHub<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FBayraktarLab%2Fcell2location%2Fdiscussions%2F133%23discussioncomment-2426344&data=04%7C01%7Ciliyal%40pitt.edu%7C68f7e7df1adb4d88afef08da0d23c128%7C9ef9f489e0a04eeb87cc3a526112fd0d%7C1%7C0%7C637836744461128839%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=0I1dz8PU%2FPK6DBW9VKHhURiYojdzQvsv2Uib2eB6kEk%3D&reserved=0>, or unsubscribe<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAANSETW4UKPDKSMNM7CBDKLVBORMXANCNFSM5RFZ5CRQ&data=04%7C01%7Ciliyal%40pitt.edu%7C68f7e7df1adb4d88afef08da0d23c128%7C9ef9f489e0a04eeb87cc3a526112fd0d%7C1%7C0%7C637836744461128839%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=EHKkqTbFYSdDwMBTooVc9JBEZEZ5XuhR7Looq2hexr4%3D&reserved=0>. You are receiving this because you authored the thread.Message ID: ***@***.***>

0 replies

adamgayoso · 2022-04-01T23:05:20Z

adamgayoso
Apr 1, 2022

cell2location depends on scvi-tools, which is causing the jax dependency. Indeed jax/jaxlib can be installed on M1 natively using pip install jax jaxlib. It's important to make sure you're using a native version of python and not a python that is being run via Rosetta. That said, scanpy cannot be installed natively on m1 because numba/llvmlite do not have m1 native wheels.

As an alternative you can install miniconda that is built for m1 macs, and then conda install scvi-tools, then pip install cell2location

1 reply

vitkl Apr 2, 2022
Maintainer

I see. Currently, cell2location is still pinned to scvi-tools <=0.14.6 so jax issue should not be relevant, right?

Also for cell2location you only need pytorch not jax.

lefterov · 2022-04-02T03:11:22Z

lefterov
Apr 2, 2022
Author

We are on the way to try something similar (more powerful GPU, though), but isn’t it a nice and a very friendly Preamble to a fashionable readthedocs user guide, where you write (humbly, but strongly), that at the time the document has been posted, the only! or the following hardware platform(s) have been tested with such and such software packages… Believe me – we will be extremely grateful to you! Nice weekend

…

-I. -- Iliya Lefterov Professor, School of Public Health University of Pittsburgh +1 412 449 9195 (mobile, USA) ***@***.*** From: Vitalii Kleshchevnikov ***@***.***> Date: Friday, April 1, 2022 at 8:21 PM To: BayraktarLab/cell2location ***@***.***> Cc: Lefterov, Iliya ***@***.***>, Author ***@***.***> Subject: Re: [BayraktarLab/cell2location] run cell2location on M1 silicon chip BookPro (Discussion #133) I see. I have not actually run the workflow on my laptop - I only used the created environments for code development in pycharm. I guess I would prefer to have a combination of simpler M1 laptops with A100 80GB GPU server to having only the best M1 laptops :) — Reply to this email directly, view it on GitHub<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FBayraktarLab%2Fcell2location%2Fdiscussions%2F133%23discussioncomment-2489196&data=04%7C01%7Ciliyal%40pitt.edu%7Cafbc3249efd84cebf94f08da143e8b08%7C9ef9f489e0a04eeb87cc3a526112fd0d%7C1%7C0%7C637844557009334644%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Zl3L%2F55HfXknjjyioWsjK%2BRqL4l38qi8MZgTN1AfKzk%3D&reserved=0>, or unsubscribe<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAANSETUZ4LXCVEIEHTBZEZ3VC6HDPANCNFSM5RFZ5CRQ&data=04%7C01%7Ciliyal%40pitt.edu%7Cafbc3249efd84cebf94f08da143e8b08%7C9ef9f489e0a04eeb87cc3a526112fd0d%7C1%7C0%7C637844557009490883%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=3JFK%2F4kCbNeCG%2B41nx3Ct3BRNI2pRGFoRxlWHHIPhXU%3D&reserved=0>. You are receiving this because you authored the thread.Message ID: ***@***.***>

0 replies

sinanugur · 2022-07-05T16:50:23Z

sinanugur
Jul 5, 2022

Hello,

I was able to install the latest pytorch v1.12 which provides GPU support on m1 but cell2location is still painfully slow. I guess it is still not optimized. Is there a way to do this or will there be an official update?


Python 3.10.5 | packaged by conda-forge | (main, Jun 14 2022, 07:05:37) [Clang 13.0.1 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.device("mps")
device(type='mps')
>>>

Cheers

5 replies

adamgayoso Jul 5, 2022

I would not expect to see massive speed improvements on the m1 cpu, but it will be supported automatically in a future release

vitkl Jul 6, 2022
Maintainer

@adamgayoso do you mean new pytorch or new scvi-tools release?

adamgayoso Jul 7, 2022

the new pytorch lightning release (we upper bound this, so then a future scvi tools release) will recognize the mps as gpu as default I believe

BenjaminDEMAILLE Jan 26, 2023

do you have some news ?

adamgayoso Jan 26, 2023

I would expect it to work with pytorch 2.0, as not all operations have been implemented to work on MPS

lefterov · 2022-07-06T02:33:46Z

lefterov
Jul 6, 2022
Author

Totally agree… put it straight: it is hard to compare the 32 cores of M1 Max GPU to 3580 cores of A30 NVIDIA GPU… some time ago I thought, because it was M1 it is always the best… I don’t think it is always the case… Best,

…

-I. -- Iliya Lefterov Professor, School of Public Health University of Pittsburgh +1 412 449 9195 (mobile, USA) ***@***.*** From: Adam Gayoso ***@***.***> Date: Tuesday, July 5, 2022 at 6:47 PM To: BayraktarLab/cell2location ***@***.***> Cc: Lefterov, Iliya ***@***.***>, Author ***@***.***> Subject: Re: [BayraktarLab/cell2location] run cell2location on M1 silicon chip BookPro (Discussion #133) I would not expect to see massive speed improvements on the m1 cpu, but it will be supported automatically in a future release — Reply to this email directly, view it on GitHub<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FBayraktarLab%2Fcell2location%2Fdiscussions%2F133%23discussioncomment-3088212&data=05%7C01%7Ciliyal%40pitt.edu%7C132b387ee0b24539808f08da5ed8623f%7C9ef9f489e0a04eeb87cc3a526112fd0d%7C1%7C0%7C637926580708289204%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=nb9skuUv2Ok77JNhnmpnsoi6A6fi445N0O8nLnTuwI8%3D&reserved=0>, or unsubscribe<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAANSETVL7677FSPK5L6HZGDVSS3RHANCNFSM5RFZ5CRQ&data=05%7C01%7Ciliyal%40pitt.edu%7C132b387ee0b24539808f08da5ed8623f%7C9ef9f489e0a04eeb87cc3a526112fd0d%7C1%7C0%7C637926580708445438%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=R5JlGsRPNuE4Cb%2BqtHtMlea%2Bpgjs9jdp%2FMgNu%2Fxc5eA%3D&reserved=0>. You are receiving this because you authored the thread.Message ID: ***@***.***>

0 replies

lefterov · 2023-01-26T18:50:22Z

lefterov
Jan 26, 2023
Author

Hi Benjamin, Unfortunately…, not and at some point we just gave up. We are guessing that working on a Linux machine with Intel(R) Xeon(R) Gold 6330 CPU @ 2.00GHz (56 cores) and NVIDIA A30 is a less frustration and incomparably faster… Best, Iliya On Jan 26, 2023, at 1:03 PM, Benjamin Demaille ***@***.******@***.***>> wrote: do you have some news ? — Reply to this email directly, view it on GitHub<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FBayraktarLab%2Fcell2location%2Fdiscussions%2F133%23discussioncomment-4790200&data=05%7C01%7Ciliyal%40pitt.edu%7Cd485da41cb8344e70e0c08daffc7a04c%7C9ef9f489e0a04eeb87cc3a526112fd0d%7C1%7C0%7C638103530114236952%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=VPTs%2BUmA1rSuDQ6j%2BIzltv4P4niGMKVKCAq%2Fy12Mnlo%3D&reserved=0>, or unsubscribe<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAANSETS3FTNDI6ZLVP7VRRLWUK367ANCNFSM5RFZ5CRQ&data=05%7C01%7Ciliyal%40pitt.edu%7Cd485da41cb8344e70e0c08daffc7a04c%7C9ef9f489e0a04eeb87cc3a526112fd0d%7C1%7C0%7C638103530114236952%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=l1znKqX0Us5g5lCMscrvroYCTnywP6mElRy0RO%2BJ4Ec%3D&reserved=0>. You are receiving this because you authored the thread.Message ID: ***@***.***>

0 replies

BenjaminDEMAILLE · 2023-01-26T18:59:10Z

BenjaminDEMAILLE
Jan 26, 2023

When PyTorch 2 will be out are you update cell2location ? What do you think of switching to tenserflow that’s already support apple silicon ?Le 26 janv. 2023 à 19:18, Adam Gayoso ***@***.***> a écrit : I would expect it to work with pytorch 2.0, as not all operations have been implemented to work on MPS —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>

1 reply

adamgayoso Jan 26, 2023

A few thoughts:

See the relevant issue here Incompatible Tensor broadcasts with MPS accelerator - Mac M2 #221
Tensorflow is outside the scope of what can be feasibly done and getting something that reproduces cell2location is a 3 month project.
There is no guarantee that cell2location will run much faster on apple GPUs. I would not expect you to see the kind of speedup you'd get from a Nvidia GPU.

abidrehman2000 · 2023-01-28T16:41:07Z

abidrehman2000
Jan 28, 2023

I believe I was having very similar issues, trying to run scvi-tools on my Apple laptop. Have you tried using mambaforge to install the packages? https://github.com/conda-forge/miniforge#mambaforge

0 replies

BenjaminDEMAILLE · 2023-01-28T17:29:13Z

BenjaminDEMAILLE
Jan 28, 2023

Pip3 Le 28 janv. 2023 à 17:41, abidrehman2000 ***@***.***> a écrit : I believe I was having very similar issues, trying to run scvi-tools on my Apple laptop. Have you tried using mambaforge to install the packages? https://github.com/conda-forge/miniforge#mambaforge —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

run cell2location on M1 silicon chip BookPro #133

{{title}}

Replies: 10 comments 13 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

run cell2location on M1 silicon chip BookPro #133

Replies: 10 comments · 13 replies

vitkl Mar 20, 2022 Maintainer

lefterov Mar 22, 2022 Author

vitkl Mar 23, 2022 Maintainer

lefterov Apr 1, 2022 Author

packages in environment at /Users/iliya/opt/anaconda3/envs/pytorch_env:

Name Version Build Channel

Name Version Build Channel

vitkl Apr 2, 2022 Maintainer

vitkl Apr 2, 2022 Maintainer

lefterov Mar 24, 2022 Author

vitkl Apr 2, 2022 Maintainer

lefterov Apr 2, 2022 Author

vitkl Jul 6, 2022 Maintainer

lefterov Jul 6, 2022 Author

lefterov Jan 26, 2023 Author

Replies: 10 comments 13 replies

vitkl
Mar 20, 2022
Maintainer

lefterov Mar 22, 2022
Author

vitkl Mar 23, 2022
Maintainer

lefterov Apr 1, 2022
Author

vitkl Apr 2, 2022
Maintainer

vitkl Apr 2, 2022
Maintainer

lefterov
Mar 24, 2022
Author

vitkl Apr 2, 2022
Maintainer

lefterov
Apr 2, 2022
Author

vitkl Jul 6, 2022
Maintainer

lefterov
Jul 6, 2022
Author

lefterov
Jan 26, 2023
Author