mOS for HPC v0.9 Readme

What is mOS for HPC

mOS for HPC is an operating systems research project at Intel, targeting extreme scale HPC systems deploying converged workflows for modeling / simulation, data analytics, and AI.  It aims to deliver a high performance computing environment with the scalability, low noise, and repeatability expected of lightweight kernels (LWK), while maintaining overall Linux compatibility that HPC plus AI / ML applications need.  

mOS for HPC remains under development at this time. These materials are being made available to interested parties to explore, to test drive, and to provide feedback through the mailing list. Consider the quality level to be pre-alpha and as-is. It is not intended to be used in production or for business critical uses. The ability to compile, install, and boot a Linux kernel is required to get mOS up and running. A good understanding of what an OS does and how it interacts with the underlying hardware is needed to configure the LWK partition and get the most out of it. Support is limited by the development team's ability to respond through the mailing list. 

What's new for v0.9?

Feature	Description	Learn More
Linux 5.10.46	mOS for HPC v0.9 is based on the long term support kernel Linux 5.10.46 from kernel.org. From a compatibility perspective, this version has been integrated and tested on a system based on SLES 15 SP2 with OpenHPC.
LWK memory manager rewrite	Major change; too many to list; Sone highlights: Code clean-up Early initializations during kernel boot Dynamic memory partitioning LWKMEM global designated memory lists per NUMA node yod option changes LWK buddy allocator More VM operations supported Page fault levels: no fault, one fault per VMR mremap support Mem policy settings via yod options Added more trace events for debugging Parallelize clearing of designated LWK memory Show LWK VMR names of LWK VMAs in maps and smaps Updated unit tests XPMEM support for new LWK memory management Randomize physical memory allocations (option for HBM cache mode)	git log
Saphire Rapids + HBM	Initial readiness
lwkctl	Can now modify only CPUs or LWKMEM, instead of both all the time	lwkctl man page
yod	Several options have changed, adding an ability disable portions of LWK memory and letting Linux handle those sections	yod man page
Scheduler and CPU management	Several enhancements: Round robin behavior changes: Better support for applications that over-commit CPUs Code clean-up and simplification Thread balancing option	git log
Bug fixes	Several, including: LWK memory regions in /proc/<pid>/smaps Phys page interleaving among NUMA domains Fix build when CONFIG_MOS_FOR_HPC not set	git log

Platform requirements

The development and testing for mOS for HPC v0.9 has been performed on systems with Intel(R) Xeon(R) Scalable processor family and on systems with Intel(R) Xeon Phi(TM) product family. As a result, mOS for HPC includes optimizations considering technologies such as multi-socket CPUs, high core counts, Intel® Hyper-Threading Technology; and complex memory configurations up to 8 NUMA domains (DDR + high bandwidth memory). Specific configurations include:

• Intel(R) Xeon(R) Platinum 8168 processors with 192GiB of DDR4, Intel(R) HT Technology on, and booted without subnuma clustering (SNC)
• Intel(R) Xeon(R) Gold 6140  processors with 128GiB of DDR4, Intel(R) HT Technology on, and booted without subnuma clustering (SNC)
• Intel(R) Xeon Phi(TM) processors 7250 with 96GiB of DRAM and 16GiB of MCDRAM, Intel(R) HT Technology on, and booted in sub-NUMA clustering 4 (SNC-4) mode and flat memory mode

Your mileage may vary on other platforms and configurations in terms of functionality and performance.

Additional remarks:

• If you use the Intel(R) Xeon Phi(TM) processor 7230, then Quadrant cluster mode, Flat memory mode is recommended.
• If you want to make all of MCDRAM available to applications on Intel(R) Xeon Phi(TM) processors, you must verify that MCDRAM is hot-pluggable in the BIOS settings.  Please see the Administrator's Guide.
• The development team has observed lower performance of mOS for HPC when running in cache memory mode on Intel(R) Xeon Phi(TM) processors, which is not necessarily attributed to hardware.
• Processors outside of the x86_64 architecture designation in Linux are unsupported – the kernel code will not configure and build.

The Linux distribution used by the development team for building, installing, and testing mOS for HPC has been for SLES 15 SP2 with OpenHPC. There also has been limited testing with CentOS 7.  Other distributions have had almost no testing, and may require adaptations for the build, install instructions to your environment.

mOS for HPC development plans to track Intel(R) oneAPI toolkits, Intel(R) Parallel Studio XE 2021 Cluster Edition for Linux*, and MPICH/MPICH4 updates as they become available. Almost no testing has been done using other compilers (e.g. gcc) or MPI runtimes (e.g. MVAPICH or OpenMPI).  

Where to get code

The mOS for HPC source can be checked out from GitHub at https://github.com/intel/mOS.   Please see the Administrator's Guide for further instructions.

Where to report issues or ask questions

Register for the mOS for HPC mailing list at https://groups.google.com/g/mos-devel/. Please, submit feedback and follow discussions through this list.

^{*Other names and brands may be claimed as the property of others.}