Skip to content

Commit

Permalink
Update to griptape 1.0 (#142)
Browse files Browse the repository at this point in the history
* Update to griptape 1.0

* Remove more engines

* Fix seo context

* Update incorrect defaults

* Update factory

* Remove engine reference

* Revert output_task change

* Revert pyproject change
  • Loading branch information
collindutter committed Dec 10, 2024
1 parent bc502df commit 986224a
Show file tree
Hide file tree
Showing 34 changed files with 2,293 additions and 2,452 deletions.
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ check: check/types check/spell check/lint ## Run all checks.

.PHONY: check/types
check/types:
@poetry run pyright docs/courses/**/code_reviews/*.py
@poetry run pyright $(shell find docs -type f -name "*.py" -path "*/code_reviews/*")

.PHONY: check/spell
check/spell:
Expand Down
10 changes: 4 additions & 6 deletions docs/courses/create-image-pipeline/02_concepts.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ In this course, we're going to be taking a topic to draw, and then execute a con

1. Use an LLM to generate an Image Generation prompt.
* If the input given is "a butterfly", the result of this task might be a text string "Create a watercolor painting of a butterfly.".
* This will allow us to ensure the output from this task will always be in the correct format for an Image Generation Engine.
* This will allow us to ensure the output from this task will always be in the correct format for an Image Generation Driver.
3. Generate an image
* Using the output from the previous step, we'll give that to an Image Generation Model
4. View the image
Expand Down Expand Up @@ -61,7 +61,7 @@ Before we dive in and start setting up our own Pipeline, it's important to revie
| **Extraction Tasks** | Various tasks associated with extracting information from text. | See examples in the [documentation](https://griptape.readthedocs.io/griptape-framework/structures/tasks/#extraction-task){target="_blank"}.
| **TextSummaryTask** | Summarizes text very efficiently | `TextSummaryTask("Imagine this is a massive amount of text.")` |
| **TextQueryTask** | Can be used to query large bodies of text, for example a vector database. | See examples in the [documentation](https://griptape.readthedocs.io/griptape-framework/structures/tasks/#text-query-task){target="_blank"} |
| **PromptImageGenerationTask** | Can be used to generate images based off a text prompt. | `PromptImageGenerationTask("watercolor butterfly"), image_generation_engine=image_engine`|
| **PromptImageGenerationTask** | Can be used to generate images based off a text prompt. | `PromptImageGenerationTask("watercolor butterfly"), image_generation_driver=image_driver`|
| **CodeExecutionTask** | Can be used to execute code. | `CodeExecutionTask(on_run=reverse_string)`|

In this course, we will be focusing mostly on **Prompt Tasks**, **Image Generation Tasks**, and **Code Execution Tasks**.
Expand All @@ -82,10 +82,8 @@ image_prompt_task = PromptTask(
#
image_generation_task = PromptImageGenerationTask(
"{{ parent_output }}", # The output of the parent task
image_generation_engine=PromptImageGenerationEngine(
image_generation_driver=OpenAiImageGenerationDriver(
model="dall-e-3", api_type="open_ai", image_size="1024x1024"
),
image_generation_driver=OpenAiImageGenerationDriver(
model="dall-e-3", api_type="open_ai", image_size="1024x1024"
),
output_dir="./images",
id="image_generation_task"
Expand Down
42 changes: 8 additions & 34 deletions docs/courses/create-image-pipeline/04_creating_images.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@
Image Generation via Griptape is handled via a few components.

* **Image Generation Driver** - Determines the model to be used. For example, [OpenAI DALL·E 3](https://openai.com/dall-e-3){target="_blank"} or [Leonardo.AI](https://leonardo.ai/){target="_blank"}.
* **Image Generation Engines** - The engine that facilitates the use of the Driver.
* **Image Generation Task** or **Image Generation Tool**. The Task or Tool is what will be provided to the Griptape Structure. Pipelines and Workflows can use `PromptImageGenerationTask` directly. You can provide an `PromptImageGenerationTool` to an Agent, to a ToolTask, or to a ToolkitTask.

For example, to create an image with OpenAI DALL·E 3 as a **task** you could do something like:
Expand All @@ -18,33 +17,28 @@ driver = OpenAiImageGenerationDriver(
model="dall-e-3", api_type="open_ai", image_size="1024x1024"
)

# Create an Image Generation Engine
engine = PromptImageGenerationEngine( image_generation_driver=driver )

# Create an Image Generation Task
task = PromptImageGenerationTask(
"Create a drawing of a pineapple",
image_generation_engine=engine,
image_generation_driver=driver,
output_dir="images"
)
```

Once you generate the task, you will add it to the pipeline or workflow.

You can also use the `PromptImageGenerationTool` tool and assign it to an Agent. It takes many of the same arguments. If you had previously created the `driver` and `engine` as specified above, you would do something like:
You can also use the `PromptImageGenerationTool` tool and assign it to an Agent. It takes many of the same arguments. If you had previously created the `driver` as specified above, you would do something like:

```python
agent = Agent(
tools=[PromptImageGenerationTool(
image_generation_engine=engine,
image_generation_driver=driver,
output_dir="images",
off_prompt=False,
)]
)
```

The main thing to be aware of is that you must use _both_ components - Driver and Engine. You choose the model with the **Driver**, use the **Engine** to facilitate the use of the model, and then access the engine with either a **Task** or a **Tool**.

In this course, because we're focusing on image generation as part of a _Pipeline_, we'll generate images using a _Task_.

## The Image Task
Expand All @@ -53,7 +47,7 @@ To get started, we'll begin by replacing the Fake Image Generation task with a r

### Imports

To use the Driver, Engine, and Task we'll need to add them to our `imports` section in `app.py`. You'll modify `griptape.tasks` to include `PromptImageGenerationTask`, and add imports for the Driver and Engine.
To use the Driver and Task we'll need to add them to our `imports` section in `app.py`. You'll modify `griptape.tasks` to include `PromptImageGenerationTask`, and add imports for the Driver and Engine.

```python hl_lines="5-7"
# ...
Expand All @@ -62,7 +56,6 @@ To use the Driver, Engine, and Task we'll need to add them to our `imports` sect
from griptape.structures import Pipeline
from griptape.tasks import PromptTask, PromptImageGenerationTask
from griptape.drivers import OpenAiImageGenerationDriver
from griptape.engines import PromptImageGenerationEngine

# ...
```
Expand All @@ -87,25 +80,6 @@ image_driver = OpenAiImageGenerationDriver(
# ...
```

### Create the Engine

The engine facilitates the use of the particular model. It will be what we pass to the task or tool. After the creation of the driver, create the engine:

```python hl_lines="8-9"
# ...

# Create the driver
image_driver = OpenAiImageGenerationDriver(
model="dall-e-3", api_type="open_ai", image_size="1024x1024"
)

# Create the engine
image_engine = PromptImageGenerationEngine(image_generation_driver=image_driver)

# Create the pipeline object
# ...
```

### Replace the ImageTask

Next, we'll replace our fake image generation task with a _real_ image generation task. Find the section of the code where we're creating the image task with `generate_image_task` and replace it with `PromptImageGenerationTask`.
Expand All @@ -115,15 +89,15 @@ Next, we'll replace our fake image generation task with a _real_ image generatio

generate_image_task = PromptImageGenerationTask(
"{{ parent_output }}",
image_generation_engine=image_engine,
image_generation_driver=image_driver,
output_dir="images",
id="Generate Image Task",
)

# ...
```

Notice we're giving it the `image_generation_engine` we defined earlier as `image_engine`. We're also specifying an `output_dir` of `images`. This will ensure the image is generated in that directory.
Notice we're giving it the `image_generation_driver` we defined earlier as `image_driver`. We're also specifying an `output_dir` of `images`. This will ensure the image is generated in that directory.

!!! tip
With the `PromptImageGenerationTask`, if you want to save the file to disk you must specify _either_ the output file name (`output_file`) or the directory you want the images to appear in (`output_dir`). If you don't, the image generated will only exist in the `ImageArtifact`.
Expand Down Expand Up @@ -250,7 +224,7 @@ If you scroll down in the code until you find the section where the `run` method
# ...

def run(self) -> ImageArtifact:
image_artifact = self.image_generation_engine.generate_image(
image_artifact = self.image_generation_driver.generate_image(
prompts=[self.input.to_text()], rulesets=self.all_rulesets, negative_rulesets=self.negative_rulesets
)

Expand Down Expand Up @@ -358,7 +332,7 @@ Now go down to the `generate_image_task` and use the `output_dir` variable in th

generate_image_task = PromptImageGenerationTask(
"{{ parent_output }}",
image_generation_engine=image_engine,
image_generation_driver=image_driver,
output_dir=output_dir,
id="Generate Image Task",
)
Expand Down
4 changes: 0 additions & 4 deletions docs/courses/create-image-pipeline/07_display_image_task.md
Original file line number Diff line number Diff line change
Expand Up @@ -168,7 +168,6 @@ from griptape.tasks import (
CodeExecutionTask
)
from griptape.drivers import OpenAiImageGenerationDriver
from griptape.engines import PromptImageGenerationEngine
from griptape.artifacts import TextArtifact

```
Expand All @@ -180,9 +179,6 @@ We won't display the image yet, we'll just create the function and make sure we
```python title="app.py" hl_lines="6-17"
# ...

# Create the engine
image_engine = PromptImageGenerationEngine(image_generation_driver=image_driver)

# Create a function to display an image
def display_image(task: CodeExecutionTask) --> TextArtifact:
# Get the filename
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@
from griptape.structures import Pipeline
from griptape.tasks import PromptTask, PromptImageGenerationTask
from griptape.drivers import OpenAiImageGenerationDriver
from griptape.engines import PromptImageGenerationEngine

load_dotenv() # Load your environment

Expand All @@ -14,8 +13,6 @@
# Create the driver
image_driver = OpenAiImageGenerationDriver(model="dall-e-3", api_type="open_ai", image_size="1024x1024")

# Create the engine
image_engine = PromptImageGenerationEngine(image_generation_driver=image_driver)

# Create the pipeline object
pipeline = Pipeline()
Expand All @@ -33,7 +30,7 @@

generate_image_task = PromptImageGenerationTask(
"{{ parent_output }}",
image_generation_engine=image_engine,
image_generation_driver=image_driver,
output_dir=output_dir,
id="Generate Image Task",
)
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
from dotenv import load_dotenv
from griptape.artifacts import TextArtifact
from griptape.drivers import OpenAiImageGenerationDriver
from griptape.engines import PromptImageGenerationEngine

# Griptape
from griptape.structures import Pipeline
Expand All @@ -19,9 +18,6 @@
# Create the driver
image_driver = OpenAiImageGenerationDriver(model="dall-e-3", api_type="open_ai", image_size="1024x1024")

# Create the engine
image_engine = PromptImageGenerationEngine(image_generation_driver=image_driver)


# Create a function to display an image
def display_image(task: CodeExecutionTask) -> TextArtifact:
Expand Down Expand Up @@ -61,7 +57,7 @@ def display_image(task: CodeExecutionTask) -> TextArtifact:

generate_image_task = PromptImageGenerationTask(
"{{ parent_output }}",
image_generation_engine=image_engine,
image_generation_driver=image_driver,
output_dir=output_dir,
id="Generate Image Task",
)
Expand Down
6 changes: 3 additions & 3 deletions docs/courses/create-image-pipeline/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,8 +49,8 @@ If you don't have those items available, it's highly recommended you go through

It's also recommended to view the [Compare Movies Workflow](../compare-movies-workflow/index.md) course if you haven't viewed it before, as it contains some similar concepts to the Pipeline course.

## Image Generation Engines
The course will cover some of the Image Generation Engines available for Griptape, including [OpenAI DALL·E 3](https://openai.com/dall-e-3){target="_blank"}, [Leonardo.AI](https://leonardo.ai/){target="_blank"}, and [Image Generation Engines running on Amazon Bedrock](https://aws.amazon.com/bedrock/stable-diffusion/){target="_blank"}
## Image Generation Drivers
The course will cover some of the Image Generation Drivers available for Griptape, including [OpenAI DALL·E 3](https://openai.com/dall-e-3){target="_blank"}, [Leonardo.AI](https://leonardo.ai/){target="_blank"}, and [Image Generation Drivers running on Amazon Bedrock](https://aws.amazon.com/bedrock/stable-diffusion/){target="_blank"}

### DALL·E 3
* DALL·E 3 is available with an OpenAI API key.
Expand All @@ -73,7 +73,7 @@ The course will cover:

* Creating Griptape Pipelines
* Creating Griptape Tasks
* Investigate Image Generation Engines
* Investigate Image Generation Drivers
* Using a `CodeExecutionTask` to display the resulting image

## Useful Resources and Links
Expand Down
6 changes: 3 additions & 3 deletions docs/courses/image-query/02_chatbot.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ from rich import print as print # Modifies print to use the Rich library
# ...
```

Then, create a function to format and print responses from the agent in dark cyan. Add this code before you start the Chat function, passing it the function with the `output_fn` parameter.
Then, create a function to format and print responses from the agent in dark cyan. Add this code before you start the Chat function, passing it the function with the `handle_output` parameter.

```python title="app.py" hl_lines="3-5 9"

Expand All @@ -77,7 +77,7 @@ def formatted_response(response: str) -> None:
# Begin Chatting
Chat(
agent,
output_fn=formatted_response, # Uses the formatted_response function
handle_output=formatted_response, # Uses the formatted_response function
).start()

```
Expand Down Expand Up @@ -114,7 +114,7 @@ Chat(
prompt_prefix="\nYou: ",
processing_text="\nThinking...",
response_prefix="\nAgent: ",
output_fn=formatted_response, # Uses the formatted_response function
handle_output=formatted_response, # Uses the formatted_response function
).start()

```
Expand Down
27 changes: 4 additions & 23 deletions docs/courses/image-query/03_image_query_tool.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,19 +26,15 @@ Add the following to your `import` section:
# ...

from griptape.tools import ImageQueryTool
from griptape.engines import ImageQueryEngine
from griptape.drivers import OpenAiImageQueryDriver
from griptape.drivers import OpenAiChatPromptDriver

# ...
```

### ImageQueryTool
This component acts as the intermediary that will manage your interactions with images. It's responsible for sending image data to the engine and retrieving results.

### ImageQueryEngine
This is where the magic happens. The engine processes the image data, analyzes it, and interprets what's in the image based on the capabilities of the underlying model.

### OpenAiImageQueryDriver
### OpenAiChatPromptDriver
This driver leverages OpenAI's advanced vision capabilities to understand and analyze images. It's a powerful tool that enables your agent to not just see images but truly understand their content, like I said before.

An example of another ImageQueryDriver is Anthropic. Check out other available drivers in the [Griptape docs](https://docs.griptape.ai/stable/griptape-framework/drivers/image-query-drivers/)
Expand All @@ -64,34 +60,19 @@ driver = OpenAiImageQueryDriver(
# ...
```

### Create the Image Query Engine

This section effectively connects our previously created driver to the engine. Think of the engine as the middle manager in your corporate structure—it doesn’t do the grunt work itself (that’s the driver’s job), but it makes sure the instructions and data flow properly to get the job done.

```python title="app.py" hl_lines="3-6"
# ...

# Create an Image Query Engine
engine = ImageQueryEngine(
image_query_driver=driver,
)

# ...
```

With the driver and engine now set up, your agent is almost ready to start describing pictures as if it were an art critic at a gallery opening. But remember, even though it can now "see" images, don't expect it to understand why your dog looks at you like that—some mysteries remain unsolved even for the most advanced AI.

Next, we'll tie all these components together and create a client that can handle all the image querying magic.

### Create the Tool

First up, let’s create and configure the `ImageQueryTool`. This component uses the engine we set up earlier to process image queries. The `off_prompt` parameter is set to `False` to ensure that the image descriptions are not just processed but also sent back to the large language model for some insightful summarization. Here's how to do it:
First up, let’s create and configure the `ImageQueryTool`. This component uses the driver we set up earlier to process image queries. The `off_prompt` parameter is set to `False` to ensure that the image descriptions are not just processed but also sent back to the large language model for some insightful summarization. Here's how to do it:

```python title="app.py" hl_lines="3-6"
# ...

image_query_tool = ImageQueryTool(
image_query_engine=engine,
prompt_driver=driver,
off_prompt=False
)

Expand Down
6 changes: 3 additions & 3 deletions docs/courses/image-query/06_querying_multiple_images.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@ Modify the code as follows:
```python title="app.py" hl_lines="5-25"
# ...
# Configure the ImageQueryTool
image_query_tool = ImageQueryTool(image_query_engine=engine, off_prompt=False)
image_query_tool = ImageQueryTool(prompt_driver=driver, off_prompt=False)

flow = "AGENT"
if flow == "WORKFLOW":
Expand All @@ -131,7 +131,7 @@ else:
prompt_prefix="\nYou: ",
processing_text="\nThinking...",
response_prefix="\nAgent: ",
output_fn=formatted_response, # Uses the formatted_response function
handle_output=formatted_response, # Uses the formatted_response function
).start()
```

Expand Down Expand Up @@ -285,7 +285,7 @@ if flow == "WORKFLOW":
+ "display the image. Save this to image_descriptions/{{ filename }}.yml\n"
+ "in YAML format.\n\n{{ parent_outputs }}",
tools=[FileManagerTool(off_prompt=False)],
context={"image": image},
context={"filename": filename},
id=f"seo_{image}",
)

Expand Down
2 changes: 1 addition & 1 deletion docs/courses/image-query/assets/code_reviews/02/app.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,5 +25,5 @@ def formatted_response(response: str) -> None:
prompt_prefix="\nYou: ",
processing_text="\nThinking...",
response_prefix="\nAgent: ",
output_fn=formatted_response, # Uses the formatted_response function
handle_output=formatted_response, # Uses the formatted_response function
).start()
Loading

0 comments on commit 986224a

Please sign in to comment.