Using Apache Flink with Clojure example

This is short demo for using Apache Flink 1.12.2 with Clojure.

Using this demo

Before using this example you need to install project prerequisites (see below).

Also, if you plan to use local cluster then download and unpack Apache Flink 1.12.2 to project folder

To run Clojure example in embedded mode use command:

just run

To build standalone app, which can be submited to Apache Flink cluster use this command:

just build

To run local Apache Flink cluster use these commands:

cd flink-1.12.2
bin/start-cluster.sh

Then uberjar may be deployed by command:

./bin/flink run target/flinkexample-0.1.0-SNAPSHOT.jar

To stop local Apache Flink cluster:

cd flink-1.12.2
bin/stop-cluster.sh

Install project prerequisites

All these tools you need to install only once.

1. Install clojure deps cli tools

   1. MacOS

      brew install clojure/tools/clojure

   2. Linux

      Ensure that the following dependencies are installed in OS: bash, curl, rlwrap, and Java.

      curl -O https://download.clojure.org/install/linux-install-1.10.2.790.sh
      chmod +x linux-install-1.10.2.790.sh
      sudo ./linux-install-1.10.2.790.sh

2. Install latest clj-new library to a file ~/.clojure/deps.edn

   {
    :aliases {
              :clj-new     {:extra-deps {seancorfield/clj-new {:mvn/version "1.1.243"}}
                            :exec-fn    clj-new/create}
             }
    }

3. Install just utility

   1. MacOS

      brew install just

   2. Linux

      wget https://github.com/casey/just/releases/download/v0.8.3/just-v0.8.3-x86_64-unknown-linux-musl.tar.gz
      
      tar -zxvf just-v0.8.3-x86_64-unknown-linux-musl.tar.gz just
      
      sudo mv ./just /usr/local/bin

4. Install babashka

   1. MacOS

      brew install borkdude/brew/babashka

   2. Linux

      sudo bash < <(curl -s https://raw.githubusercontent.com/babashka/babashka/master/install)

5. Run once:

   just requirements

   to install other necessary tools (MacOS only, for Linux manual instruction).

6. Run once in project folder:

   direnv allow

to allow direnv to load environment variables from .env.public and .env.private files.

Project workflow

To configure project workflow scripts use Justfile.

To configure project version use file version_id

To configure group-id and artifact-id for jar file use file .env.public

Run just to show help for project workflow.

List of available just commands:

• requirements - install necessary tools (MacOS only): clj-kondo, cljstyle, direnv. For Linux install them manually.

• clean - clear target folder

• repl - run Clojure REPL

• javac - compile java sources (if any)

• build - build uberjar file (application)

• install - install uberjar file to local .m2

• outdated - check for outdated dependencies

• deploy - deploy uberjar file to remote repository (stub)

• test - run tests

• bump - bump version artifact in version_id file, level may be one of: major, minor, patch, alpha, beta, rc, release.

• format - format source code

• lint - lint source code

• standalone - create a standalone application with bundled JDK (using jlink, JDK 9+)

License

Copyright © 2021 Mike Ananev
Distributed under the Eclipse Public License 2.0 or (at your option) any later version.