Add macros to run the examples in the Exunit test suite

README.md

@@ -1,6 +1,11 @@
 # ExExample
 
-**TODO: Add description**
+`ExExample` aims to provide an example-driven test framework for Elixir applications.
+
+As opposed to regular unit tests, examples are supposed to be executed from within the REPL.
+
+Examples serve both as a unit test, but also as a tool to discover, learn, and interact with a live
+system such as Elixir applications.
 
 ## Installation
 
@@ -15,7 +20,43 @@ def deps do
 end
 ```
 
-Documentation can be generated with [ExDoc](https://github.com/elixir-lang/ex_doc)
-and published on [HexDocs](https://hexdocs.pm). Once published, the docs can
-be found at <https://hexdocs.pm/ex_example>.
+## Your First Example
+
+To get started, create a new module in the `lib/` folder of your Elixir application and add an example.
+
+```elixir
+defmodule MyExamples do
+  use ExExample
+  defexample read_data() do
+    1..1000 |> Enum.shuffle() |> Enum.take(10)
+  end
+end
+```
+
+In a running REPL with your application loaded, you can execute this example using `MyExamples.read_data()`.
+The example will be executed once, and the cached result will be returned the next time around.
+
+## Caching
 
+In a REPL session it's not uncommon to recompile your code (e.g., using `recompile()`). This changes
+the semantics of your examples.
+
+To avoid working with stale outputs, `ExExample` only returns the cached version of your example
+if the code it depends on, or the example itself, have not been changed.
+
+When the code changes, the example is executed again.
+
+## Tests 
+
+The examples are created to work with the code base, but they can also serve as a unit test. 
+
+To let ExUnit use the examples in your codebase as tests, add a test file in the `test/` folder, and
+import the `ExExample.Test` module. 
+
+To run the examples from above, add a file `ny_examples_test.exs` to your `test/` folder and include the following.
+
+```elixir
+defmodule MyExamplesTest do
+  use ExExample.Test, for: MyExamples
+end
+```

lib/ex_example.ex

@@ -14,4 +14,45 @@ defmodule ExExample do
   def start(_type, args \\ []) do
     ExExample.Supervisor.start_link(args)
   end
+
+  @doc """
+  I am the use macro for ExExample.
+
+  I import the ExExample.Behaviour module, expose the macros, and define the `copy/1` and `rerun?/1` callbacks.
+  """
+  defmacro __using__(_options) do
+    quote do
+      import unquote(ExExample.Macro)
+      # module attribute that holds all the examples
+      Module.register_attribute(__MODULE__, :examples, accumulate: true)
+
+      # import the behavior for the callbacks
+      @behaviour ExExample.Behaviour
+
+      # default implementation of the callbacks
+      def copy(result) do
+        result
+      end
+
+      def rerun?(_result) do
+        false
+      end
+
+      # mark the callbacks are overridable.
+      defoverridable copy: 1
+      defoverridable rerun?: 1
+
+      @before_compile unquote(__MODULE__)
+    end
+  end
+
+  defmacro __before_compile__(_env) do
+    quote do
+      def run_examples() do
+        Enum.each(@examples, fn example ->
+          apply(__MODULE__, example, [])
+        end)
+      end
+    end
+  end
 end

lib/ex_example/analyze.ex

@@ -0,0 +1,147 @@
+defmodule ExExample.Analyze do
+  @moduledoc """
+  I contain functions that help analyzing modules and their dependencies.
+
+  I have functionality to extract a list of modules that are being called by a module, as well
+  as a function to calculate a hash for a module and its dependencies.
+  """
+
+  @doc """
+  I return a hash for the given module and its dependencies.
+
+  When any of these dependencies are recompiled, this hash will change.
+  """
+  @spec dependencies_hash(atom() | String.t() | tuple()) :: integer()
+  def dependencies_hash(module) do
+    module
+    |> dependencies()
+    |> Enum.map(& &1.module_info())
+    |> :erlang.phash2()
+  end
+
+  @doc """
+  I analyze the module and return a list of all the modules it calls.
+  I accept a module name, a piece of code as string, or an AST.
+  """
+  @spec dependencies(atom() | String.t() | tuple()) :: MapSet.t(atom())
+  def dependencies(module) when is_atom(module) do
+    case get_in(module.module_info(), [:compile, :source]) do
+      nil ->
+        []
+
+      source ->
+        to_string(source)
+        |> File.read!()
+        |> dependencies()
+    end
+  end
+
+  def dependencies(module) when is_binary(module) do
+    module
+    |> Code.string_to_quoted()
+    |> dependencies()
+  end
+
+  def dependencies(module) when is_tuple(module) do
+    deps_for_module(module)
+  end
+
+  @doc """
+  I extract all the modules that the given AST calls.
+
+  Aliases that are not used are ignored.
+  """
+  @spec deps_for_module(Macro.t()) :: MapSet.t(atom())
+  def deps_for_module(ast) do
+    # extract all the alias as expressions
+    {_, deps} =
+      Macro.postwalk(ast, %{}, fn
+        # a top-level alias. E.g., `alias Foo.Bar, as: Bloop`
+        # {:alias, [line: 3], [{:__aliases__, [line: 3], [:Bloop]}, [as: {:__aliases__, [line: 3], [:Bar]}]]}
+        ast = {:alias, _, [{:__aliases__, _, aliases}, [as: {:__aliases__, _, [as_alias]}]]}, acc ->
+          # canonicalize the alias atoms
+          aliases = Enum.map(aliases, &Module.concat([&1]))
+          as_alias = Module.concat([as_alias])
+
+          # check if the root has been aliased, replace if so
+          [root | rest] = aliases
+          root = Map.get(acc, root, root)
+          aliases = [root | rest]
+
+          # if the first atom is an alias, resolve it
+          module = Module.concat(aliases)
+          {ast, Map.put(acc, as_alias, module)}
+
+        # alias erlang module. E.g., `alias :code, as: Code`
+        ast = {:alias, _, [module, [as: {:__aliases__, _, [as_alias]}]]}, acc when is_atom(module) ->
+          as_alias = Module.concat([as_alias])
+          {ast, Map.put(acc, as_alias, module)}
+
+        # a top-level alias. E.g., `alias Foo.Bar`
+        # {:alias, [line: 2], [{:__aliases__, [line: 2], [:X, :Y]}]}
+        ast = {:alias, _, [{:__aliases__, _, aliases}]}, acc ->
+          # canonicalize the alias atoms
+          aliases = Enum.map(aliases, &Module.concat([&1]))
+
+          # check if the root has been aliased, replace if so
+          [root | rest] = aliases
+          root = Map.get(acc, root, root)
+          aliases = [root | rest]
+
+          # store the alias chain
+          module = Module.concat(aliases)
+          aliased = List.last(aliases)
+          {ast, Map.put(acc, aliased, module)}
+
+        # top-level group alias. E.g., `alias Foo.{Bar, Baz}`
+        # {:alias, [line: 2], [{:__aliases__, [line: 2], [:X, :Y]}]}
+        ast = {{:., _, [{:__aliases__, _, aliases}, :{}]}, _, sub_alias_list}, acc ->
+          # canonicalize the alias atoms
+          aliases =
+            Enum.map(aliases, &Module.concat([&1]))
+
+          # check if the root is an alias
+          # check if the root has been aliased, replace if so
+          [root | rest] = aliases
+          root = Map.get(acc, root, root)
+          aliases = [root | rest]
+
+          # resolve the subaliases
+          acc =
+            for {:__aliases__, _, sub_aliases} <- sub_alias_list, into: acc do
+              sub_aliases = Enum.map(sub_aliases, &Module.concat([&1]))
+              aliased_as = List.last(sub_aliases)
+              {aliased_as, Module.concat(aliases ++ sub_aliases)}
+            end
+
+          {ast, acc}
+
+        # function call to module. E.g., `Foo.func()`
+        # {:alias, [line: 2], [{:__aliases__, [line: 2], [:X, :Y]}]}
+        ast = {{:., _, [{:__aliases__, _, aliases}, _func]}, _, _args}, acc ->
+          # canonicalize the alias atoms
+          aliases = Enum.map(aliases, &Module.concat([&1]))
+
+          # check if the root is an alias
+          # check if the root has been aliased, replace if so
+          [root | rest] = aliases
+          root = Map.get(acc, root, root)
+          aliases = [root | rest]
+
+          # canonicalize the alias atoms
+          module = Module.concat(aliases)
+          {ast, Map.update(acc, :calls, MapSet.new([module]), &MapSet.put(&1, module))}
+
+        # the module itself is included in the dependencies
+        ast = {:defmodule, _, [{:__aliases__, _, module_name}, _]}, acc ->
+          module_name = Module.concat(module_name)
+          acc = Map.update(acc, :calls, MapSet.new([module_name]), &MapSet.put(&1, module_name))
+          {ast, acc}
+
+        ast, acc ->
+          {ast, acc}
+      end)
+
+    Map.get(deps, :calls, [])
+  end
+end

lib/ex_example/cache/cache.ex

@@ -0,0 +1,37 @@
+defmodule ExExample.Cache do
+  @moduledoc """
+  I define logic to store and retrieve results from the cache.
+  """
+  alias ExExample.Cache.Key
+  alias ExExample.Cache.Result
+
+  require Logger
+
+  @cache_name :ex_examples
+
+  @doc """
+  I store a result in cache for a given key.
+  """
+  @spec store_result(Result.t(), Key.t()) :: {atom(), boolean()}
+  def store_result(%Result{} = result, %Key{} = key) do
+    Logger.debug("store result for #{inspect(key)}: #{inspect(result)}")
+    Cachex.put(@cache_name, key, result)
+  end
+
+  @doc """
+  I fetch a previous Result from the cache if it exists.
+  If it does not exist, I return `{:error, :not_found}`.
+  """
+  @spec get_result(Key.t()) :: {:ok, any()} | {:error, :no_result}
+  def get_result(%Key{} = key) do
+    case Cachex.get(@cache_name, key) do
+      {:ok, nil} ->
+        Logger.debug("cache miss for #{inspect(key)}")
+        {:error, :no_result}
+
+      {:ok, result} ->
+        Logger.debug("cache hit for #{inspect(key)}")
+        {:ok, result}
+    end
+  end
+end

lib/ex_example/cache/cache_key.ex

@@ -0,0 +1,19 @@
+defmodule ExExample.Cache.Key do
+  @moduledoc """
+  I represent the key for an example invocation.
+
+  I identify an invocation by means of its module, name, arity, and list of arguments.
+  """
+  use TypedStruct
+
+  typedstruct enforce: true do
+    @typedoc """
+    I represent the key for an example invocation.
+    """
+    field(:deps_hash, integer())
+    field(:module, atom())
+    field(:name, String.t() | atom())
+    field(:arity, non_neg_integer(), default: 0)
+    field(:arguments, list(any()), default: [])
+  end
+end

lib/ex_example/cache_result.ex → lib/ex_example/cache/cache_result.ex

@@ -1,18 +1,17 @@
-defmodule ExExample.CacheResult do
+defmodule ExExample.Cache.Result do
   @moduledoc """
   I represent the cached result of a ran Example
   """
 
+  alias ExExample.Cache.Key
+
   use TypedStruct
 
-  typedstruct enforce: true do
+  typedstruct enforce: false do
     @typedoc """
     I represent the result of a completed Example Computation
     """
-
-    field(:source, Macro.input())
-    field(:source_name, atom())
+    field(:key, Key.t())
     field(:result, term())
-    field(:pure, boolean())
   end
 end

lib/ex_example/examples/e_cache_result.ex

@@ -0,0 +1,55 @@
+defmodule ExExample.Examples.ECacheResult do
+  @moduledoc """
+  I am an example module that demonstrates how to use the `defexample` macro.
+
+  There are two examples defined: `read_data/0` and `process_data/0`.
+
+  The `read_data/0` example generates some random data each time it is executed.
+
+  The `process_data/0` example reads the data generated by `read_data/0` and processes it.
+
+  If the examples are run, the result of `read_data/0` and `process_data/0` will be remembered.
+  Only when any of its dependencies change, the examples will be re-executed.
+
+  The optional `rerun?/1` and `copy/1` callbacks are defined to control when to re-run an example or
+  how to copy the output of an example's previous run.
+  """
+  use ExExample
+
+  @doc """
+  I am an example that does some intense computations we don't want to repeat.
+  """
+  defexample read_data() do
+    1..1000 |> Enum.shuffle() |> Enum.take(10)
+  end
+
+  @doc """
+  I process some data
+  """
+  defexample process_data() do
+    data = read_data()
+
+    IO.puts("processing the data")
+    data
+  end
+
+  @doc """
+  I get called whenever an example is executed.
+  I can return true if the given result must be re-evaluated, or false when the cached value can
+  be used.
+
+  I am optional.
+  """
+  def rerun?(_result) do
+    false
+  end
+
+  @doc """
+  I copy the result of a previous execution.
+
+  I am optional.
+  """
+  def copy(result) do
+    result
+  end
+end