Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update docs site for version 2.18.1 #71

Closed
wants to merge 1 commit into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
3 changes: 3 additions & 0 deletions versioned_docs/version-2.18/docs/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
node_modules
package.json
package-lock.json
60 changes: 60 additions & 0 deletions versioned_docs/version-2.18/docs/NOTES.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
# Docs process

Pants currently hosts documentation at Readme.com, and we use a combination of their `rdme` tool to sync handwritten markdown docs, and a custom `generate-docs` script to update Pants' reference documentation.

Currently the rdme process is manual, until we bed down the process, at which point we'll add it to CI.

The motivation for in-repo docs is covered [on this Google doc](https://docs.google.com/document/d/1bZE8PlF9oRzcPQz4-JUFr5vfD0LFHH4V3Nj2k221CFM/view)

## Versions

Readme expects every version of the docs to correspond to a semver release. Our convention is as follows:

- A version on readme.com corresponds to a pants release (e.g. pants `v2.11` has docs `v2.11`)
- The current development (`main` branch) docs are kept in a readme.com version that will reflect the next version of Pants (e.g. if the most recent release branch is `v2.97`, then `main`'s docs should be synced to `v2.98`).

# Using `rdme` (general notes)

## Setup

### Install `node`

```
brew install node
```

### Install `rdme`

From the `docs` directory,

```
npm install rdme
```

### Log in.

```
npx rdme login --project pants
```

(`rdme` will prompt for two-factor-authentication codes if necessary)

## When cutting a new release branch

Create a fork of the most recent docs branch, and mark it as `beta`, for example:

```
npx rdme versions:create --version=v2.98 --fork="v2.97" --main=false --beta=true --isPublic=true
```

will create a new docs version, `2.98` based on a copy of the docs from version `2.97`.

## Sync docs changes up to `readme.com`

Docs markdown files are stored in the `markdown` directory. `rdme` does not do bidirectional sync, so any changes made on readme.com itself _will be deleted_. Make sure you apply any changes from readme.com locally before syncing up.

From the root of the repository:

```
npx rdme docs docs/markdown --version v2.98
```
4 changes: 0 additions & 4 deletions versioned_docs/version-2.18/docs/ad-hoc-tools/_category_.json

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

4 changes: 0 additions & 4 deletions versioned_docs/version-2.18/docs/docker/_category_.json

This file was deleted.

This file was deleted.

18 changes: 0 additions & 18 deletions versioned_docs/version-2.18/docs/getting-started/index.mdx

This file was deleted.

63 changes: 0 additions & 63 deletions versioned_docs/version-2.18/docs/getting-started/prerequisites.mdx

This file was deleted.

4 changes: 0 additions & 4 deletions versioned_docs/version-2.18/docs/go/_category_.json

This file was deleted.

This file was deleted.

10 changes: 0 additions & 10 deletions versioned_docs/version-2.18/docs/go/integrations/index.mdx

This file was deleted.

4 changes: 0 additions & 4 deletions versioned_docs/version-2.18/docs/helm/_category_.json

This file was deleted.

4 changes: 0 additions & 4 deletions versioned_docs/version-2.18/docs/introduction/_category_.json

This file was deleted.

This file was deleted.

Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
---
title: Integrating new tools without plugins
sidebar_position: 999
title: "Integrating new tools without plugins"
slug: "adhoc-tool"
hidden: false
createdAt: "2023-03-20T00:00:00.000Z"
---

---

## Integrating new tools without plugins
# Integrating new tools without plugins

The `adhoc_tool` target allows you to execute "runnable" targets inside the Pants sandbox. Runnable targets include first-party sources that can be run with `pants run`, 3rd-party dependencies like `python_requirement` or `jvm_artifact`, or even executables that exist on your system and managed externally to Pants.

`adhoc_tool` provides you with the building blocks needed to put together a custom build process without needing to develop and maintain a plugin. The level of initial effort involved in using `adhoc_tool` is significantly lower than that of [writing a plugin](../writing-plugins/plugins-overview.mdx), so it's well-suited to consuming one-off scripts, or for rapidly prototyping a process before actually writing a plugin. The tradeoff is that there is more manual work involved in defining build processes that reflect your codebase's structure, and that the targets that define the tools you consume are less easy to reuse.
`adhoc_tool` provides you with the building blocks needed to put together a custom build process without needing to develop and maintain a plugin. The level of initial effort involved in using `adhoc_tool` is significantly lower than that of [writing a plugin](doc:plugins-overview), so it's well-suited to consuming one-off scripts, or for rapidly prototyping a process before actually writing a plugin. The tradeoff is that there is more manual work involved in defining build processes that reflect your codebase's structure, and that the targets that define the tools you consume are less easy to reuse.

The `antlr` demo in the [`example-adhoc` respository](https://github.com/pantsbuild/example-adhoc) shows an example of running a JVM-based tool to transparently generate Python code that can be used in another language:

Expand All @@ -28,21 +28,21 @@ adhoc_tool(
)
```

### `runnable` targets
## `runnable` targets

"Runnable" targets are targets that Pants knows how to execute within its sandbox. Generally, these correspond to targets that can be executed with the `pants run` goal, and include first-party source files, as well as third-party dependencies.

The tool will be run with values from `args` specified as arguments. By default, the process' working directory will be the directory where the `BUILD` file is defined. This can be adjusted using the `workdir` field.

:::caution `runnable` targets must be pure functions
When run by `adhoc_tool`, Pants assumes that the inputs provided to the process -- that is, the values of the `adhoc_tool`'s fields, and the contents of the runnable and execution dependencies -- fully describe the output. Output values will be [cached](../introduction/how-does-pants-work.mdx#caching) by Pants, and future invocations with identical inputs will be retrieved from the cache instead of being re-executed. If your process has behavior that is not fully defined by its inputs, Pants' behavior may be unexpected or inconsistent.
:::
> 🚧 `runnable` targets must be pure functions
>
> When run by `adhoc_tool`, Pants assumes that the inputs provided to the process -- that is, the values of the `adhoc_tool`'s fields, and the contents of the runnable and execution dependencies -- fully describe the output. Output values will be [cached](doc:how-does-pants-work#caching) by Pants, and future invocations with identical inputs will be retrieved from the cache instead of being re-executed. If your process has behavior that is not fully defined by its inputs, Pants' behavior may be unexpected or inconsistent.

:::caution `runnable` targets must be idempotent
`adhoc_tool` processes may be cancelled or retried any number of times, so it is important that any side effects are idempotent. That is, it should not matter if it is run several times, or only partially.
:::
> 🚧 `runnable` targets must be idempotent
>
> `adhoc_tool` processes may be cancelled or retried any number of times, so it is important that any side effects are idempotent. That is, it should not matter if it is run several times, or only partially.

### Specifying dependencies
## Specifying dependencies

`adhoc_tool` has more complexity surrounding dependencies compared with Pants' first-class targets. This is because you need to do manual work to set up the execution environment, which is usually taken care of by plugin code.

Expand All @@ -54,31 +54,31 @@ When run by `adhoc_tool`, Pants assumes that the inputs provided to the process

In the `antlr` example, `output_dependencies` is used because the tool produces Python-based bindings that depend on a runtime library. `execution_dependencies` specifies the sources that are consumed by the tool, but do not need to be consumed by subsequent build steps.

### Specifying outputs
## Specifying outputs

Generally, `adhoc_tool` targets are run to produce outputs that can be supplied to other targets. These can be in the form of files or directories that are output directly by the tools: use the `output_files` field to capture individual files, or `output_directories` to capture entire directories as output.

Files are captured relative to the build root by default: this is useful when passing results to further `adhoc_tool` targets defined in the same `BUILD` file. If this behavior is not right for you, for example, if you are producing an artifact for packaging, you can change the root of the outputs using the `root_output_directory` field.

Finally, if you want to capture `stdout` or `stderr` from your tool, you can use the `stdout` or `stderr` fields. These specify filenames where those streams will be dumped once the process completes. Note that these files are specified in addition to those from the `output_files` field, and an error will occur if the filename occurs in the outputs arising from `output_files` or `output_directories` and the contents of that file are different.

### Chaining processes together
## Chaining processes together

_Our [JavaScript demo](https://github.com/pantsbuild/example-adhoc/tree/main/javascript) demonstrates a string of `adhoc_tool` targets that's used to produce a resource file._

To get the best cache efficiency, it can make sense to break your `adhoc_tool` into smaller incremental steps. For example, if your process needs to fetch dependencies and then build a library based on those dependencies and some first-party source files, having one `adhoc_tool` for each of those steps means that the dependency-fetching stage will only be re-run when your requirements change, and not when the first-party source files change.

Generally, if you are chaining `adhoc_tool` targets together , it will be easier to use the default `workdir` and `root_output_directory` fields for each step that will be consumed by an `adhoc_tool` in the same `BUILD` file. Change the `root_output_directory` only for targets that are intended to be used in other places or ways.

### Wrapping generated sources for use by other targets
## Wrapping generated sources for use by other targets

_Our [Antlr demo](https://github.com/pantsbuild/example-adhoc/tree/main/antlr) demonstrates wrapping the outputs of `adhoc_tool` targets for use as Python sources._

`adhoc_tool` generates `file` sources by default. This can be acceptable if generating assets that do not need to be consumed as source files for another Pants backend. Other Pants backends need generated sources to be marked as actual source files.

There are several targets included in Pants with the prefix `experimental_wrap_as_`. These act as a source target that can be used as a dependency in a given language backend, with the caveat that dependency inference is not available.

### Using externally-managed tools
## Using externally-managed tools

_Our [JavaScript demo](https://github.com/pantsbuild/example-adhoc/tree/main/javascript) demonstrates the use of externally-managed binaries._

Expand All @@ -88,7 +88,7 @@ Some build processes need to make use of tools that can't be modeled within a Pa

When specified as a `runnable_dependency`, the binary will be available on the `PATH` with the target name of the dependency. This can be important if the `runnable` field invokes a subprocess (for example, `yarn` tries to invoke a binary called `node` as its interpreter).

### Running shell scripts
## Running shell scripts

Currently, `shell_source` targets are not runnable. In the meantime, it is possible to run a shell script as an `adhoc_tool` through the following approach:

Expand Down
Loading