Document developing on the devstack Open edX event bus.

README.rst

@@ -20,8 +20,8 @@ Documentation
 Development Workflow
 --------------------
 
-One Time Setup
-~~~~~~~~~~~~~~
+Local Development Set Up
+~~~~~~~~~~~~~~~~~~~~~~~~
 .. code-block::
 
   # Clone the repository
@@ -43,9 +43,24 @@ One Time Setup
   # Run edx-exams locally
   python manage.py runserver localhost:18740 --settings=edx_exams.settings.local
 
+Devstack Set Up
+~~~~~~~~~~~~~~~
+.. code-block::
+
+  # Clone the repository
+  git clone [email protected]:edx/edx-exams.git
+  cd edx-exams
 
-Every time you develop something in this repo
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+  # Start LMS in devstack from your local devstack directory
+  make dev.up.lms
+
+  # Return to the edx-exams repo directory and provision the edx-exams containers
+  bash provision-edx-exams.sh
+
+You can use the make targets defined in the ``Makefile`` to interact with the running ``edx-exams`` Docker containers.
+
+Development Workflow
+~~~~~~~~~~~~~~~~~~~~
 .. code-block::
 
   # Activate the virtualenv
@@ -79,6 +94,88 @@ Every time you develop something in this repo
 
   # Open a PR and ask for review.
 
+Event Bus Set Up
+~~~~~~~~~~~~~~~~
+
+The ``edx-exams`` service uses the Open edX event bus to publish events relating to the exam attempt lifecycle and
+others important exam events. These Open edX events are emitted by the service and pushed onto the event bus. Downstream
+services, like the LMS, receive these events and implement downstream effects of these events. For more details,
+please see `Implementation of Event Driven Architecture for Exam Downstream Effects`_.
+
+These focus of these instructions is on how to set up the Open edX event bus for use with ``edx-exams``. For more
+documentation about the event bus in general, please see `How to start using the Event Bus`_.
+
+Currently, the event bus is only supported in environments running Docker containers, like `devstack`_. This is because
+the interactions between services on the event bus is implemented in the devstack networking layer.
+
+In order to run the event bus locally, follow these steps. These steps assume that you both have `devstack`_ running and
+that you are running the ``edx-exams`` Docker container, as described in the Devstack Set Up section. These steps
+describe how to install and run the Kafka-based event bus.
+
+1. In a ``requirements/private.txt`` file, add the following Python package. These requirements are necessary for the
+   Kafka-based event bus. They are not included as a part of the standard set of requirements because installation of
+   confluent_kafka poses issues for users of Tutor on M1 Macs, which includes many users in the Open edX community. 
+   For more details, please see `Optional Import of Confluent Kafka`_.
+
+
+  .. code-block::
+
+    confluent_kafka[avro,schema-registry]
+
+2. Install the application requirements to install ``confluent_kafka``.
+
+  .. code-block::
+
+    # Shell into the application Docker container
+    make app-shell
+
+    # Install requirements
+    make requirements
+
+3. Follow the `manual testing`_ instructions to set up the Kafka-based Open edX event bus in the service that contains
+   the event handler(s) for your event(s) - for example, the LMS or Studio.
+
+Producing Events
+################
+
+Events will be produced at key stages of the exam attempt lifecycle and other points in the special exam feature. If you
+are using the local Kafka cluster, you will be able to see the topics and events there.
+
+Consuming Events
+################
+
+In order to consume events off the event bus, you must run a management command that starts an infinite loop to read
+from the event bus.
+
+Shell into the application Docker container and run the following management command to start the loop. See the
+`consume_events management command documentation`_ for a description of the arguments.
+
+.. code-block::
+
+  python3 manage.py consume_events -t <topic-name> -g <group-id>
+
+Here is an example of a command to consume events from the ``learning-exam-attempt-lifecycle`` topic in the LMS.
+
+.. code-block::
+
+    python3 manage.py ls consume_events -t learning-exam-attempt-lifecycle -g dev-lms
+
+When your event is successfully consumed, you should see logs like the following.
+
+.. code-block::
+
+  2023-10-04 15:50:17,508 INFO 554 [edx_event_bus_kafka.internal.consumer] [user None] [ip None] consumer.py:513 - Message received from Kafka: topic=dev-learning-exam-attempt-lifecycle, partition=0, offset=7, message_id=b71c735c-62cd-11ee-9064-0242ac120012, key=b'\x00\x00\x00\x00\x010course-v1:edX+777+2023FW', event_timestamp_ms=1696434617498
+
+  2023-10-04 15:50:17,593 INFO 554 [edx_event_bus_kafka.internal.consumer] [user None] [ip None] consumer.py:393 - Message from Kafka processed successfully
+
+
+.. _Implementation of Event Driven Architecture for Exam Downstream Effects: https://github.com/edx/edx-exams/blob/main/docs/decisions/0004-downstream-effect-events.rst
+.. _How to start using the Event Bus: https://openedx.atlassian.net/wiki/spaces/AC/pages/3508699151/How+to+start+using+the+Event+Bus
+.. _devstack: https://edx.readthedocs.io/projects/open-edx-devstack/en/latest/
+.. _Optional Import of Confluent Kafka: https://github.com/openedx/event-bus-kafka/blob/main/docs/decisions/0005-optional-import-of-confluent-kafka.rst.
+.. _manual testing: https://github.com/openedx/event-bus-kafka/blob/main/docs/how_tos/manual_testing.rst
+.. _consume_events management command documentation: https://github.com/openedx/openedx-events/blob/7e6e92429485133bf16ae4494da71b5a2ac31b9e/openedx_events/management/commands/consume_events.py
+
 Setting up an exam and proctoring tool
 --------------------------------------
 

docker-compose.yml

@@ -1,7 +1,9 @@
 version: "2.1"
 services:
   db:
-    image: mysql:5.6
+    # Oracle-packaged version includes a `linux/arm64/v8` version, needed for
+    # machines with Apple Silicon CPUs (Mac M1, M2)
+    image: mysql:8.0.33-oracle
     container_name: edx_exams.db
     environment:
       MYSQL_ROOT_PASSWORD: ""