Sequence background transpiler

[goetzrrGit]

• Tickets addressed: Improve large expansion sets #1324
• Review: By commit
• Merge strategy: Merge (no squash)

Description

Part 2 of 2 for #1324

You need PR #1324 merged first

This PR introduces a background transpiler process to help improve the upfront cost of creating an expansion set or sequence expansion

Changes:

• Fetches the latest command dictionary, mission model, and expansion logic every 5 minutes
• Transpiles and caches the results

Verification

Manual testing with Clipper mission model and command dictionary

[cohansen]

I see your comment about merging this into the other PR, this was just an easier first review since it was more digestible.

[dyst5422]

Just want to ask a perhaps dumb question. This essentially polls every 5 minutes. Is there any way we can change from a pull to a push model? IE be notified of changes and trigger transpilation, but throttle it to every 5 minutes? That way, no cycles are wasted pulling and transpiling when there are no changes?

[goetzrrGit]

@dyst5422 I like the idea of combining these two concepts. We can trigger transpilation whenever the user uploads a new command dictionary or mission model. The current polling mechanism can help address edge cases where users operate outside the sequencing server and Hasura GraphQL Actions. This typically occurs when users bypass the /put-expansion endpoint, often in cases like Clipper, which heavily relies on the API and not Aerie UI.

@dandelany Do you want me to explore this option, or create a ticket to address this later? I want to get these new changes to Clipper to try it out and see if it helps them in the thread test.

[dyst5422]

@dyst5422 I like the idea of combining these two concepts. We can trigger transpilation whenever the user uploads a new command dictionary or mission model. The current polling mechanism can help address edge cases where users operate outside the sequencing server and Hasura GraphQL Actions. This typically occurs when users bypass the /put-expansion endpoint, often in cases like Clipper, which heavily relies on the API and not Aerie UI.

Sorry for the ignorance, how does use of the api by clipper bypass /put-expansion?

[goetzrrGit]

@dyst5422 Not at all, I noticed Aerie UI was doing this. When the user creates a new authoring logic it does not use the /put-expansion endpoint. It is using a different GQL call which bypasses the Hasura action and server. I am sure Clipper or other users can do the same. The problem is the user will always do something different than intended so the pull is a good way to catch those edge-cases. I think the hybrid model will be good which will have the responsiveness of the push model but keep the pull model as a backup for the time the user does something they are not supposed to.

I talked to @dandelany and @joswig about this yesterday and we were going to make a ticket to add a push model to the code at a later time. We had some important EDSL tickets come from clipper that me and Matt need to prototype.

[dyst5422]

@dyst5422 Not at all, I noticed Aerie UI was doing this. When the user creates a new authoring logic it does not use the /put-expansion endpoint. It is using a different GQL call which bypasses the Hasura action and server. I am sure Clipper or other users can do the same. The problem is the user will always do something different than intended so the pull is a good way to catch those edge-cases. I think the hybrid model will be good which will have the responsiveness of the push model but keep the pull model as a backup for the time the user does something they are not supposed to.

I talked to @dandelany and @joswig about this yesterday and we were going to make a ticket to add a push model to the code at a later time. We had some important EDSL tickets come from clipper that me and Matt need to prototype.

It's concerning to me that there are multiple paths to get an expansion in the system as we clearly have processes we want to execute as part of the paths through and don't have control over that flow. Is there interest in closing those alternate paths to getting an expansion into the system?

100% support the appropriate prioritization.

[goetzrrGit]

We will address the UI hole, which should resolve most of the issues. However, since missions/users have visibility into the Hasura CLI, they can still see all available GQL and potentially bypass our recommended queries. This is a very unlikely situation, but users are going to do weird things especially if they are outside of Aerie.

[goetzrrGit]

Create a new ticket to create a push model.

#1349

[goetzrrGit]

Going to revise this code when I work on #1349