Skip to content

Commit

Permalink
extend editor docs
Browse files Browse the repository at this point in the history
  • Loading branch information
capital-G committed Sep 26, 2023
1 parent 5da18c2 commit 519d98d
Show file tree
Hide file tree
Showing 6 changed files with 196 additions and 23 deletions.
Binary file added docs/assets/editor.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/front-popup.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
161 changes: 143 additions & 18 deletions docs/editor.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,31 +3,110 @@
Caster Editor
=============

.. contents:: :local:
:depth: 2


.. seealso::

This documents covers the editing of a *story graph*.
For the concepts of such a graph see :ref:`story-graph`.
For the concepts of such a graph see :ref:`Story Graph`.

The editor is a website which allows to create and modify a :ref:`Story Graph`.
It is possible that many users at once can collaboratively edit this story graph, and also the modifications will be applied in real time to an already running stream, which allows for live coding setups.

.. figure:: ./assets/editor.png
:alt: A screenshot of the editor

A screenshot of the editor which shows all the *nodes* of a story graph as well as their connections, called *edges*.
If a node is double clicked, the node editor will be opened (right side) which shows the :ref:`editor script cells` and :ref:`editor node doors` of the node,
whose content can be edited here.

.. _caster-editor-script-cells:
.. _editor script cells:

Script cells
------------

Script cells contain different kind of instructions which control what happens on the stream or the story graph.
There are multiple types of script cells for different contexts.

.. list-table:: Script cells
:widths: 10 30
:header-rows: 1

* - Script cell type
- Description
* - Markdown
- Write text which will be transferred to a spoken text via a Text to Speech engine.
The text can also be variable by using the value of a :ref:`Stream Variable`.
* - Audio
- Provides an interface to upload prepared audio files so that they can be played back on the stream.
* - SuperCollider
- SuperColider is a framework for audio engine and Gencaster uses it to generate the audio stream.=
It also contains the scripting language *sclang* which allows to adjust the auditory content on the stream.
* - Python
- Python is a scripting language which allows to modify and interact with the story graph and also trigger
e.g. PopUps on the :ref:`caster-front`.
Anything that should be decided dynamically (e.g. day vs night) should be coded within Python.

Markdown
^^^^^^^^

Markdown is a way to write formatted text in an easy but extensible manner.
The written text will be converted to audio via :class:`~stream.models.TextToSpeech`.

It is possible to make this text dynamic by using the value of a :class:`~stream.models.StreamVariable` within the text.
It is possible to make this text dynamic by using the value of a :ref:`Stream Variable` within the text.

All custom Gencaster markdown macros are documented in :class:`~story_graph.markdown_parser.GencasterRenderer`.

Examples
""""""""

**Speak a text on the stream**

By creating a markdown script cell and enter the text

.. code-block:: markdown
Ich wünsch einen schönen Tag.
the text *Ich wünsche einen schönen Tag* will be spoken on the stream.

**Add a break between words**

.. code-block:: markdown
Hallo {break}`300ms` dort.
**Switch between voices**

There is a variety of voces to choose from, which are documented in :class:`stream.models.TextToSpeech.VoiceNameChoices`.

The default voice is ``DE_NEURAL2_C__FEMALE``, but if we want to switch temporary to e.g. a male voice it is possible via

.. code-block:: markdown
This is me and {male}`this is also another me`.
.. todo::

Switch between multiple speakers by their name.


**Use the name of the user**

First it is necessary to trigger a popup in the frontend so that the user can enter their name.
This is handled via a Python script cell, see :ref:`trigger a popup via a python script cell <python ask name>` which stores the name under the :ref:`Stream Variable` ``name``.

To use this ``name`` within a Markdown cell can be archived via


.. code-block:: markdown
Hello ${var}`name`.
I hope you are doing fine.
Hello ${var}`name`. I hope you are doing fine.
where the *${var}`name* will be replaced with the name provided through the popup, so for example *Hello Alice. I hope you are doing fine*.

All possible macros are documented in :class:`~story_graph.markdown_parser.GencasterRenderer`.

Audio
^^^^^
Expand All @@ -37,34 +116,60 @@ An audio cell allows to playback a :class:`~stream.models.AudioFile` on the stre
- ``async`` will playback in the background and the story graph will continue with the execution
- ``sync`` will pause the execution of the story graph until the file has been played back fully.

The *volume* slider controls the volume of the audio on the stream.

The *edit* button allows to change the associated audio file by uploading a new file or search through existing files.

Python
^^^^^^

Python is an universal scripting language and allows to

- interact with the graph (e.g. set next node)
- create or read out :class:`~stream.models.StreamVariable`
- assign or access a :ref:`Stream Variable`
- trigger dialogs on the frontend

The execution is async which allows to wait for something to happen, e.g. for a stream variable through an user interaction with the help of :meth:`~story_graph.engine.Engine.wait_for_stream_variable`.
Examples
""""""""

**Wait until user clicks on "Start" button**

When a user visits the story graph via the :ref:`Frontend <caster-front>` a first popup will be displayed which asks if the user wants to start streaming audio.
This is necessary due to `Autoplay restrictions in browsers <https://developer.mozilla.org/en-US/docs/Web/Media/Autoplay_guide>`_ which require a user interaction to playback any audio.

So in order to wait until the user hears audio (which will happen after the user clicks on start) the following snippet can be used.

.. code-block:: python
await wait_for_stream_variable('start')
A stream variable can be accessed through a dictionary
The execution is async which allows to wait for something to happen, for example waiting for a :ref:`Stream Variable` like in this case.
For more technical details see :meth:`~story_graph.engine.Engine.wait_for_stream_variable`.

**Access a stream variable**

A :ref:`Stream Variable` can be accessed through the ``vars`` dictionary.

.. code-block:: python
# will return None if not set
name = vars.get('name')
In order to set a stream variable, simply set the value in the ``vars`` dictionary
**Set a stream variable**

Assigning a :ref:`Stream Variable` to a value is possible by using the ``vars`` dictionary

.. code-block:: python
vars['day'] = 8 < datetime.now().hour < 20
vars['is_day'] = 8 < datetime.now().hour < 20
.. important::

Although this statements results in a boolean value, a stream variable can only represent a string as it gets shared with many languages.

.. _python ask name:

**Create popup which asks for the name**

To trigger a dialog in the frontend use the ``yield`` command with a :class:`~stream.frontend_types.Dialog` instance.
The exact arguments are documented in :ref:`frontend-types`.
Expand All @@ -84,29 +189,49 @@ The exact arguments are documented in :ref:`frontend-types`.
This will save the value that was inserted by the user into tho variable ``name``.

.. figure:: ./assets/front-popup.png
:alt: Popup in the front end which asks for the name of the user.
:scale: 50%

The code above generates a pop up where the user is asked for his name.
The reply from the user is stored in the stream variable ``name``.


SuperCollider
^^^^^^^^^^^^^

A SuperCollider cell allows to directly control what happens in the audio domain of the stream.
The language used for this is *sclang* and the dialect of choice is the *JITlib* dialect.
The language used for this is *sclang* and the dialect of choice is the `*JITlib* dialect <https://doc.sccode.org/Overviews/JITLib.html>`_, although any other style is also possible, but may require more management.

Although *sclang* is a non-blocking language, it is possible to ``wait`` within a Script Cell and it is the easiest way to wait some time.

The code can be async and will be handled in blocking manner which allows to wait on the stream, e.g. for a fade-in to finish.
**Wait 10 seconds**

The following snippet will fade in over 10 seconds and after this will fade out within 1 second.

.. code-block:: supercollider
Ndef(\drone, {
SinOsc.ar(LFDNoise1.kr(0.1!2).exprange(100, 400)) * \amp.kr(0.2);
}).fadeTime_(10.0).play;
10.0.wait;
.. important::

Do not execute ``Tdef.clear`` as this will remove the beacon of the server and
the stream will be considered offline from there on.
Ndef(\drone).stop(fadeTime: 1.0);
Comment
^^^^^^^

A comment will be ignored for execution, but allows to add commentary on a node.

.. _editor node doors:

Node doors
----------


Meta
----


Debug Streams
-------------
2 changes: 1 addition & 1 deletion docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ The audio streams have a low latency (about 150ms) and can be listened to in any

* :ref:`caster-front` is a web frontend that allows users to listen to the streams of caster-sound, written in Vue

* :ref:`caster-editor` is a web-editor in which the actions of a stream, called a :ref:`story-graph`, can be created, edited and tweaked using Python or SuperCollider, and is also written in Vue
* :ref:`caster-editor` is a web-editor in which the actions of a stream, called a :ref:`Story Graph`, can be created, edited and tweaked using Python or SuperCollider, and is also written in Vue

.. toctree::
:maxdepth: 2
Expand Down
20 changes: 20 additions & 0 deletions docs/services.rst
Original file line number Diff line number Diff line change
@@ -1,6 +1,26 @@
.. _services:

Services
========

Gencaster consists of multiple services which cover various tasks around the concept of a :ref:`Story Graph`.

.. list-table:: Services
:header-rows: 1
:widths: 10 30

* - Name
- Description
* - :ref:`Frontend <caster-front>`
- Interact and stream audio of a story graph
* - :ref:`Backend <caster-back>`
- Stores information about streams and story graphs and manages their coordination
* - :ref:`Editor <caster-editor>`
- Create and edit story graphs within the browser
* - :ref:`Sound <caster-sound>`
- Creates an audio stream which gets distributed via WebRTC


.. toctree::
:maxdepth: 3
:caption: Contents:
Expand Down
36 changes: 32 additions & 4 deletions docs/story_graph.rst
Original file line number Diff line number Diff line change
@@ -1,8 +1,12 @@
.. _story-graph:
.. _Story Graph:

Story Graph
===========

.. contents:: :local:
:depth: 2


.. seealso::

This documents covers the concept behind a story graph.
Expand All @@ -20,10 +24,10 @@ A story graph allows to construct such possibilities which cover
and much more.

Node
^^^^
----

A *Node* represents a group of statements which will be executed for the listener upon visiting said node.
Each node contains one or multiple *script cells*, which are described further at :ref:`caster-editor-script-cells`.
Each node contains one or multiple *script cells*, which are described further at :ref:`editor script cells`.

Each session of a listener starts on the *Start node*.

Expand All @@ -36,7 +40,7 @@ Each session of a listener starts on the *Start node*.


Edge
^^^^
----

In order to jump from one node to another one it is necessary to connect these two with an *edge*.
This *edge* is directed, which means that the edge works only in one direction, from an outgoing *start node*
Expand Down Expand Up @@ -75,3 +79,27 @@ It is possible to create loops within the story graph.
"Play music" -> "Play podcast";
"Play podcast" -> "Start";
}

.. _Node Door:

Node Door
---------

For a more technical description of a Node Door see :class:`story_graph.models.NodeDoor`.

.. _Stream Variable:

Stream Variable
---------------

For a more technical description of a Stream Variable see :class:`stream.models.StreamVariable`.

.. _Graph Meta:

Graph Metadata
--------------

Stream assignment policy
^^^^^^^^^^^^^^^^^^^^^^^^

There are...

0 comments on commit 519d98d

Please sign in to comment.