Many-Core Engine for Perl

This document describes MCE version 1.900.

Many-Core Engine (MCE) for Perl helps enable a new level of performance by maximizing all available cores.

Description

MCE spawns a pool of workers and therefore does not fork a new process per each element of data. Instead, MCE follows a bank queuing model. Imagine the line being the data and bank-tellers the parallel workers. MCE enhances that model by adding the ability to chunk the next n elements from the input stream to the next available worker.

Synopsis

This is a simplistic use case of MCE running with 5 workers.

 # Construction using the Core API

 use MCE;

 my $mce = MCE->new(
    max_workers => 5,
    user_func => sub {
       my ($mce) = @_;
       $mce->say("Hello from " . $mce->wid);
    }
 );

 $mce->run;

 # Construction using a MCE model

 use MCE::Flow max_workers => 5;

 mce_flow sub {
    my ($mce) = @_;
    MCE->say("Hello from " . MCE->wid);
 };

The following is a demonstration for parsing a huge log file in parallel.

 use MCE::Loop;

 MCE::Loop->init( max_workers => 8, use_slurpio => 1 );

 my $pattern  = 'something';
 my $hugefile = 'very_huge.file';

 my @result = mce_loop_f {
    my ($mce, $slurp_ref, $chunk_id) = @_;

    # Quickly determine if a match is found.
    # Process the slurped chunk only if true.

    if ($$slurp_ref =~ /$pattern/m) {
       my @matches;

       # The following is fast on Unix, but performance degrades
       # drastically on Windows beyond 4 workers.

       open my $MEM_FH, '<', $slurp_ref;
       binmode $MEM_FH, ':raw';
       while (<$MEM_FH>) { push @matches, $_ if (/$pattern/); }
       close   $MEM_FH;

       # Therefore, use the following construction on Windows.

       while ( $$slurp_ref =~ /([^\n]+\n)/mg ) {
          my $line = $1; # save $1 to not lose the value
          push @matches, $line if ($line =~ /$pattern/);
       }

       # Gather matched lines.

       MCE->gather(@matches);
    }

 } $hugefile;

 print join('', @result);

The next demonstration loops through a sequence of numbers with MCE::Flow.

 use MCE::Flow;

 my $N = shift || 4_000_000;

 sub compute_pi {
    my ( $beg_seq, $end_seq ) = @_;
    my ( $pi, $t ) = ( 0.0 );

    foreach my $i ( $beg_seq .. $end_seq ) {
       $t = ( $i + 0.5 ) / $N;
       $pi += 4.0 / ( 1.0 + $t * $t );
    }

    MCE->gather( $pi );
 }

 # Compute bounds only, workers receive [ begin, end ] values

 MCE::Flow->init(
    chunk_size  => 200_000,
    max_workers => 8,
    bounds_only => 1
 );

 my @ret = mce_flow_s sub {
    compute_pi( $_->[0], $_->[1] );
 }, 0, $N - 1;

 my $pi = 0.0;  $pi += $_ for @ret;

 printf "pi = %0.13f\n", $pi / $N;  # 3.1415926535898

Installation and Dependencies

To install this module type the following:

MCE_INSTALL_TOOLS=1 perl Makefile.PL   (to include bin/mce_grep)
(or)                perl Makefile.PL

make
make test
make install

This module requires Perl 5.8.1 or later to run. By default, MCE spawns threads on Windows and child processes otherwise on Cygwin and Unix platforms. The use of threads requires that you include threads support prior to loading MCE.

processes   (or)   use threads;           (or)   use forks;
                   use threads::shared;          use forks::shared;
use MCE;           use MCE;                      use MCE;

MCE utilizes the following modules, which are mostly installed with Perl:

bytes
constant
Carp
Errno
Fcntl
File::Path
IO::Handle
Scalar::Util
Sereal::Decoder 3.015+ (optional)
Sereal::Encoder 3.015+ (optional)
Socket
Storable   2.04+ (default when Sereal isn't available)
Test::More 0.45+ (for make test only)
Time::HiRes

Further Reading

The Perl MCE module is described at https://metacpan.org/pod/MCE.

MCE options are described at metacpan. It includes several demonstrations at the end of the page.

See also, MCE::Examples.

Copyright and Licensing

Copyright (C) 2012-2024 by Mario E. Roy

This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself:

    a) the GNU General Public License as published by the Free
    Software Foundation; either version 1, or (at your option) any
    later version, or

    b) the "Artistic License" which comes with this Kit.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See either the GNU General Public License or the Artistic License for more details.

You should have received a copy of the Artistic License with this Kit, in the file named "LICENSE". If not, I'll be glad to provide one.

You should also have received a copy of the GNU General Public License along with this program in the file named "Copying". If not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA or visit their web page on the internet at https://www.gnu.org/copyleft/gpl.html.