RFC: System Buildpacks #179

jkutner · 2021-07-29T03:27:02Z

This RFC supplements #175 and #168

buildpack-bot · 2021-07-29T03:27:16Z

Maintainers,

As you review this RFC please queue up issues to be created using the following commands:

/queue-issue <repo> "<title>" [labels]...
/unqueue-issue <uid>

Issues

✅ b56ab4 - [RFC #0101] Implement System Buildpacks #209 "Implement System Buildpacks"

sclevine · 2021-07-30T02:55:32Z

text/0000-system-buildpacks-in-builder.md

+
+## Detection
+
+The results of detection by system buildpacks MUST NOT influence the selected buildpack group. If no system buildpacks pass detection, any buildpack group MAY pass detection. If a system buildpack pass detection and no buildpack groups pass detection, then detection MUST fail.


The results of detection by system buildpacks MUST NOT influence the selected buildpack group.

I'm not sure I understand this requirement. If system buildpacks can augment the build plan, they will also influence the selected buildpack group.

If no system buildpacks pass detection, any buildpack group MAY pass detection.

Similar issue to the above -- certain groups may require a system buildpack to include build plan entries.

If a system buildpack pass detection and no buildpack groups pass detection, then detection MUST fail.

Is this just to prevent groups with all optional buildpacks from turning into groups that only contain system buildpacks?

The results of detection by system buildpacks MUST NOT influence the selected buildpack group.

I'm not sure I understand this requirement. If system buildpacks can augment the build plan, they will also influence the selected buildpack group.

I really meant for this to be about exit code. With regard to the build plan, maybe the system buildpack should have either not have that, or have a separate build plan.

I think that I would like to add a +1 to @sclevine point. I don't understand why these wouldn't just be treated as buildpacks that you want to run before and after every other group. Maybe I am missing something bigger here but I feel like the concept to me was simpler if it is essentially just matrix expansion if you will. And with the ability to mark the buildpacks optional I think you still have a very wide range of flexibility.

I'm hoping to avoid having our decision to decouple certain behaviors (like .profile) from the implementation of the lifecycle manifest itself in the user experience.

I think buildpack users shouldn't need to understand or care how things like "optional" or groups, or any other buildpack concepts work. They should run pack build (or whatever for their platform) and it just detects a python app, and builds it. Adding buildpacks like .profile or whatever into the build breaks the abstraction provided by pack. (also maybe this decision can be scoped to pack and not lifecycle).

Ultimately though, I think this comes down to how much we decide to push into the system/utility buildpacks. If it's really just .profile in most cases, then I probably don't care as much. But I worry about a situation where it used to be as simple as "the heroku/python buildpack passed detect" to "why are there 8 buildpacks running?" Again I think that comes back to a difference b/w Heroku/Paketo buildpack design.

sclevine · 2021-07-30T02:58:17Z

text/0000-system-buildpacks-in-builder.md

+
+## Build
+
+System buildpacks that have passed detection will be executed during the build phase. All `pre` buildpacks must execute before the detected buildpack group. All `post` buildpacks must execute after the detected buildpack group. However, the system buildpacks MUST NOT be added to the buildpack group, and their execution may be hidden from the end user.


Why do we want to hide system buildpacks from the user? Shouldn't we be transparent about what features are activated?

the system buildpacks MUST NOT be added to the buildpack group

Not sure I understand what this means.

Why do we want to hide system buildpacks from the user? Shouldn't we be transparent about what features are activated?

I went back and forth on this. But the fact that they are buildpacks is a bit of an implementation detail. In verbose logging you would want to see this, but in the normal flow of things, people want to see the buildpacks they care about "node, java, etc". This may be a difference in the Heroku/Paketo stance too.

the system buildpacks MUST NOT be added to the buildpack group

Not sure I understand what this means.

This is really about exit code again, and not wanting passing system buildpacks to allow groups with only optional buildpacks that fail to continue. I'll reword

I can see both ways on this. If we're really extracting out spec/lifecycle features we didn't necessarily log that .profile was written.

Would system buildpacks be a part of https://github.com/buildpacks/spec/blob/main/platform.md#iobuildpacksbuildmetadata-json?

hone · 2021-08-11T15:42:53Z

text/0000-system-buildpacks-in-builder.md

+
+The results of detection by system buildpacks MUST NOT influence the selected buildpack group. If no system buildpacks pass detection, any buildpack group MAY pass detection. If a system buildpack pass detection and no buildpack groups pass detection, then detection MUST fail.
+
+System buildpacks may require/provide in the build plan following standard buildpack API specification.


Do you have use cases for system buildpacks you're thinking of that would use the Build Plan? I think with how much you want to "hide" system buildpacks if it's bad that they'd influence the other buildpack execution.

I have no specific examples/use-cases. I only wanted to default to the having them be the "same" as normal buildpacks

I am not so sure on including system buildpacks in the buildplan if they are not interacting with the normal buildpacks in a group. Could we either limit the system buildpacks plan resolution to their respective groups? pre system buildpacks need to match up all their plan entries separately, normal buildpacks separately and post buildpacks separately.

Given that we want to discourage people from using system buildpacks as hidden buildpacks and given that these are mainly supposed to be utility buildpacks - would a simplifying assumption be to remove system buildpacks from contributing to the build plan entirely? And limit them to just detect pass or fail?

Could we either limit the system buildpacks plan resolution to their respective groups?

Meaning they have their own plan, distinct from user defined buildpacks? I think that's a good idea.

I'm fine with any of these, but I agree that I should probably be more specific about which behavior we want in the RFC.

Should we just have buildpacks not be part of the buildplan for now if we don't have a use case for it?

hone · 2021-08-11T15:52:30Z

@jkutner is it out of scope where the system buildpacks come from? They don't live on the registry do they?

jkutner · 2021-08-11T16:07:57Z

@hone i suppose it's out of scope, but I could see them on the registry.

hone · 2021-09-15T18:24:59Z

From last WG, this is blocked on the resolution of #175

jkutner · 2021-12-03T13:33:30Z

@hone @sclevine please review

hone

It looks like there are two remaining issues that still need to be resolved before we can move this forward:

system buildpacks' Build Plan - https://github.com/buildpacks/rfcs/pull/179/files#r686954199
system buildpacks being hidden from user - https://github.com/buildpacks/rfcs/pull/179/files#r679613177

jkutner · 2022-01-20T14:06:51Z

@hone please review the stance taken in the text:

system buildpacks' Build Plan

system buildpacks use the build plan as normal buildpacks do

system buildpacks being hidden from user

They "may" be hidden. I think we should leave this up to platforms.

jkutner · 2022-01-27T15:27:35Z

From working group:

Enforce that system buildpacks are always optional. Remove optional from schema (but this might cause a problem if we ever want to introduce optional, because the default would change)
spec changes will require a new .toml or similar in the builder
lifecycle will need a new flag --system-buildpacks
need to define how this behaves with different platform API that doesn't support (recommend: it should warn)

text/0000-system-buildpacks-in-builder.md

ekcasey · 2022-02-03T14:56:14Z

text/0000-system-buildpacks-in-builder.md

+
+## Use the `[[order]]` table
+
+Instead of a new `[system]` table, we could put `pre` and `post` in the `[[order]]` table. However, this could imply that there is a interaction/override/etc between these buildpacks and the `pre`/`post` buildpacks in `project.toml`. But there is not.


I kind of like this alternative builder.toml schema? But it's not a strong preference.

I don't have a strong preference

text/0000-system-buildpacks-in-builder.md

Signed-off-by: Joe Kutner <[email protected]> Co-authored-by: Emily Casey <[email protected]>

Signed-off-by: Joe Kutner <[email protected]>

text/0000-system-buildpacks-in-builder.md

Signed-off-by: Joe Kutner <[email protected]>

Signed-off-by: Joe Kutner <[email protected]> Co-authored-by: Emily Casey <[email protected]>

jkutner · 2022-02-04T21:00:10Z

Moving this into FCP, closing Feb 11. (next Friday)

hone · 2022-03-02T18:40:14Z

@buildpacks/platform-contributors @buildpacks/implementation-maintainers are there any issues that should be created for this?

ekcasey · 2022-03-04T19:04:47Z

/queue-issue buildpacks/rfcs "Implement System Buildpacks"

[#179] Signed-off-by: Emily Casey <[email protected]>

jkutner requested a review from a team as a code owner July 29, 2021 03:27

jkutner force-pushed the jkutner/system-buildpacks branch from 0e26318 to b94cb2f Compare July 29, 2021 03:28

jkutner changed the title ~~Add RFC for System Buildpacks~~ RFC: System Buildpacks Jul 29, 2021

sclevine reviewed Jul 30, 2021

View reviewed changes

nebhale requested review from nebhale, ekcasey and hone August 6, 2021 17:11

nebhale approved these changes Aug 6, 2021

View reviewed changes

hone reviewed Aug 11, 2021

View reviewed changes

jkutner added the spec/platform label Aug 12, 2021

ekcasey added the team/implementation label Aug 25, 2021

github-actions bot assigned nebhale Aug 25, 2021

ekcasey added the audience/buildpack-author label Aug 27, 2021

hone added the status/blocked RFC is blocked by some other outstanding dependency. label Sep 15, 2021

sclevine approved these changes Dec 3, 2021

View reviewed changes

hone reviewed Dec 8, 2021

View reviewed changes

hone removed the status/blocked RFC is blocked by some other outstanding dependency. label Dec 8, 2021

jkutner requested a review from hone January 20, 2022 14:04

hone approved these changes Jan 26, 2022

View reviewed changes

jkutner force-pushed the jkutner/system-buildpacks branch from 5af5178 to e5dad2f Compare February 2, 2022 20:21

ekcasey requested changes Feb 3, 2022

View reviewed changes

ekcasey requested a review from sclevine February 3, 2022 15:06

jkutner force-pushed the jkutner/system-buildpacks branch from 2e17f87 to eac4899 Compare February 3, 2022 17:23

jkutner and others added 7 commits February 4, 2022 08:20

Update text/0000-system-buildpacks-in-builder.md

87c01ba

Signed-off-by: Joe Kutner <[email protected]> Co-authored-by: Emily Casey <[email protected]>

Update text/0000-system-buildpacks-in-builder.md

d77ef97

Signed-off-by: Joe Kutner <[email protected]> Co-authored-by: Emily Casey <[email protected]>

Update text/0000-system-buildpacks-in-builder.md

daed18e

Signed-off-by: Joe Kutner <[email protected]> Co-authored-by: Emily Casey <[email protected]>

Update text/0000-system-buildpacks-in-builder.md

1ec097e

Signed-off-by: Joe Kutner <[email protected]> Co-authored-by: Emily Casey <[email protected]>

Update text/0000-system-buildpacks-in-builder.md

3d9d225

Signed-off-by: Joe Kutner <[email protected]> Co-authored-by: Emily Casey <[email protected]>

add system.toml for detect

51bd0d9

Signed-off-by: Joe Kutner <[email protected]>

udpates to allow optional system buildpacks

e7a6d9a

Signed-off-by: Joe Kutner <[email protected]>

jkutner force-pushed the jkutner/system-buildpacks branch from c3e315e to e7a6d9a Compare February 4, 2022 14:21

ekcasey reviewed Feb 4, 2022

View reviewed changes

fix spec in system buildpacks

eb7217e

Signed-off-by: Joe Kutner <[email protected]>

ekcasey approved these changes Feb 4, 2022

View reviewed changes

jkutner and others added 5 commits February 4, 2022 14:52

Update text/0000-system-buildpacks-in-builder.md

e269b13

Signed-off-by: Joe Kutner <[email protected]> Co-authored-by: Emily Casey <[email protected]>

Update text/0000-system-buildpacks-in-builder.md

e03e627

Signed-off-by: Joe Kutner <[email protected]> Co-authored-by: Emily Casey <[email protected]>

Update text/0000-system-buildpacks-in-builder.md

48d1da2

Signed-off-by: Joe Kutner <[email protected]> Co-authored-by: Emily Casey <[email protected]>

Update text/0000-system-buildpacks-in-builder.md

e32acbb

Signed-off-by: Joe Kutner <[email protected]> Co-authored-by: Emily Casey <[email protected]>

Merge branch 'main' into jkutner/system-buildpacks

a918b93

sclevine approved these changes Feb 4, 2022

View reviewed changes

jkutner added the Final Comment Period label Feb 4, 2022

Merge branch 'main' into jkutner/system-buildpacks

487b43c

hone assigned ekcasey and unassigned nebhale Mar 2, 2022

buildpack-bot mentioned this pull request Mar 4, 2022

[RFC #0101] Implement System Buildpacks #209

Open

8 tasks

ekcasey added a commit that referenced this pull request Mar 4, 2022

RFC 0101

c6c99c7

[#179] Signed-off-by: Emily Casey <[email protected]>

ekcasey merged commit 31c14c3 into main Mar 4, 2022

ekcasey deleted the jkutner/system-buildpacks branch March 4, 2022 19:20

ekcasey removed the Final Comment Period label Mar 4, 2022

ekcasey removed their assignment Mar 4, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RFC: System Buildpacks #179

RFC: System Buildpacks #179

jkutner commented Jul 29, 2021 •

edited

Loading

buildpack-bot commented Jul 29, 2021 •

edited

Loading

sclevine Jul 30, 2021

jkutner Jul 30, 2021

ForestEckhardt Jul 30, 2021

jkutner Jul 30, 2021

sclevine Jul 30, 2021

jkutner Jul 30, 2021

hone Aug 11, 2021

hone Dec 8, 2021

hone Aug 11, 2021 •

edited

Loading

jkutner Aug 11, 2021

sambhav Dec 3, 2021

jkutner Dec 6, 2021

hone Dec 8, 2021

hone commented Aug 11, 2021

jkutner commented Aug 11, 2021

hone commented Sep 15, 2021

jkutner commented Dec 3, 2021

hone left a comment

jkutner commented Jan 20, 2022

jkutner commented Jan 27, 2022 •

edited

Loading

ekcasey Feb 3, 2022

jkutner Feb 3, 2022

jkutner commented Feb 4, 2022

hone commented Mar 2, 2022 •

edited

Loading

ekcasey commented Mar 4, 2022


		## Detection

		The results of detection by system buildpacks MUST NOT influence the selected buildpack group. If no system buildpacks pass detection, any buildpack group MAY pass detection. If a system buildpack pass detection and no buildpack groups pass detection, then detection MUST fail.


		## Build

		System buildpacks that have passed detection will be executed during the build phase. All `pre` buildpacks must execute before the detected buildpack group. All `post` buildpacks must execute after the detected buildpack group. However, the system buildpacks MUST NOT be added to the buildpack group, and their execution may be hidden from the end user.


		The results of detection by system buildpacks MUST NOT influence the selected buildpack group. If no system buildpacks pass detection, any buildpack group MAY pass detection. If a system buildpack pass detection and no buildpack groups pass detection, then detection MUST fail.

		System buildpacks may require/provide in the build plan following standard buildpack API specification.


		## Use the `[[order]]` table

		Instead of a new `[system]` table, we could put `pre` and `post` in the `[[order]]` table. However, this could imply that there is a interaction/override/etc between these buildpacks and the `pre`/`post` buildpacks in `project.toml`. But there is not.

RFC: System Buildpacks #179

RFC: System Buildpacks #179

Conversation

jkutner commented Jul 29, 2021 • edited Loading

buildpack-bot commented Jul 29, 2021 • edited Loading

Issues

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hone Aug 11, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hone commented Aug 11, 2021

jkutner commented Aug 11, 2021

hone commented Sep 15, 2021

jkutner commented Dec 3, 2021

hone left a comment

Choose a reason for hiding this comment

jkutner commented Jan 20, 2022

jkutner commented Jan 27, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jkutner commented Feb 4, 2022

hone commented Mar 2, 2022 • edited Loading

ekcasey commented Mar 4, 2022

jkutner commented Jul 29, 2021 •

edited

Loading

buildpack-bot commented Jul 29, 2021 •

edited

Loading

hone Aug 11, 2021 •

edited

Loading

jkutner commented Jan 27, 2022 •

edited

Loading

hone commented Mar 2, 2022 •

edited

Loading