Low Code Plugin Programming Model

[MikeStall]

This memo describes a proposal to finalize the programming model for Low Code plugins.

Low Code plugins run as a C# Plugin which includes the power fx nugets and binds maker expressions into the dataverse eventing pipeline.

This is the proposed GA behavior; so it has some changes from current preview behavior.
See migration steps below. Notable breaking changes from current:

• usage of Set vs. Patch in automated plugins . Set will modify the "current" entity; Patch will call onto IOrgnizationService.
• avoiding ThisRecord because it conflicts with builtin ThisRecord used in PowerApps' functions like Filter/LookUp. Instead use a new identifier NewRecord. Instead of XPrevRecord(), use OldRecord.
• requiring tables to be explicitly added in the UI before they can be used.
• no implicit scope for accessing fields. Can't say Field2, but instead say NewRecord.Field2'. This applies to Set as well: we're require Set(NewRecord.Field1, OldRecord.Field2*10)`

Core scenarios

Low Code plugins can be triggered as both Custom APIs ("Instant") and Entity messages ("Automated").

For "Instant Plugins" (aka APIs):

Triggered: Explicitly invoke as an unbound action

1. Receive input parameters and return output parameters.

For "automated plugins"

Triggered: registered on an entity's messsage (Create/Update/Delete/etc) and Stage (pre-op/post-op/etc)

1. Update fields of the current entity
2. Validation rules, especially on create and update.
3. Setting Default values on create
4. Logging and auditing
5. Block duplicate entries

Principles

1. Align to dataverse "physics" - Low Code plugins are implemented on top of Dataverse's existing Plugin infrastructure and eventing system (https://learn.microsoft.com/en-us/power-apps/developer/data-platform/event-framework). Power Fx operations should clearly align to the existing concepts.
2. Align to Power Apps. - Makers should be able to copy and paste expressions from Power Apps into a low code plugin. This effectively moving code from client-side to server side.
3. Versioning - Designs must be safe when changes occur. For example, new fields can be added to a dataverse table; or new tables can be added. See below for more details.

Versioning Concerns

Expressions must not change meaning in the future. For example, if an expression is written on monday, and a new field is added to a Dataverse on tuedsay, the expression can't rebind to that new field in a way that would change semantics.

Consider the following kinds of changes that can happen in a dataverse environment:

1. new fields can be added to an existing Dataverse table;
2. new tables can be added.
3. new builtin functions can be added to Power Fx.
4. new objects could be added. Today, we have User, NewRecord, OldRecord, etc... but future identifiers can be added.
5. Display names can be renamed. Display names can also conflict.

Step to ensure versioning:

1. Serialize as logical names, not display names. On editing, the editor can convert logical names to the current display names (and disambiguate in case of conflict), and then convert back to logical names on save. More generally - the principle is to serialize in a canonical unambiguous form.
2. No implicit scope. Only a few key identifiers are in top-level scope, such as NewRecord. An expression can bind to NewRecord.Field1, but not to Field1 directly.
3. Explicit references - Tables (and other references) must be explicitly enabled. This means if a new table is added in dataverse, the expression will not even see it (or be able to bind to it) unless the maker takes action to add it to the list of table references.
4. Closest definition wins.
5. It's ok for a previous invalid expression to now become valid. But not for a valid expression to change meaning. Hence, err on the side of making something an error - since that gives us freedom to resolve it later.

Common Features

All Power Fx Language Features

Log Code plugins uses the C# interpreter implementation of Power Fx, which enables all language features, including:

• With()
• RegEx support, although in preview pending Regular Expression syntax needs to be normalized across Power Fx back ends #1817.
• Json / ParseJSON() support.

This is more feature rich than the Power Fx Formula Columns - which have the restriction of compiling to SQL. (see https://learn.microsoft.com/en-us/power-apps/maker/data-platform/formula-columns)

No implicit scope and top-level objects.

Any object at top-level scope must be explicitly specified. This is especially important for versioning issues - we don't want an expression to change meaning if you add new fields.

Builtin top-level objects

1. Environment object.
2. User object.

Table References

Any table reference must be explicitly added in the UI.
For example, in order for the expression to Collect(Accounts, {... }) , the Accounts table needs to be added in the UI.
This helps with intellisense - Collect/Patch intellisense can suggest a short list of tables.
It also is critical for versioning - adding new tables in the future does not change the expression's semantics.

Additional objects.

For instant plugins, the input parameters are accessible as top-level objects.

For automated plugins, there is NewRecord and OldRecord. Fields on current entity are not directly accessible and must be accessed like NewRecord.Field1.

Validation Errors

If a plugin returns a runtime error, such as by using the Error() function, then it will block the operation. This can be used to implement validation rules.

Validation rules should be done in pre-op.

If( NewRecord.HotelCost + NewRecord.TravelCost < NewRecord.AllowedBudget
    Error( "Total cost is over budget" }
)

See https://learn.microsoft.com/en-us/power-platform/power-fx/reference/function-iferror for more details on Power Fx error functions.

Access Dataverse Tables

Plugins can read and write to dataverse tables. This includes Collect(), Patch(), Filter(), LookUp().
Delegation operations are supported and a warning is issued if an expression can't be delegated.

These operations will run directly against the Plugin's IOrganizationService (not the current entity) and will directly operate on the database. They will run in the transaction context.

Variables

Plugins do not support contexts, named formulas, or variables. Whereas Power Apps lets makers use Set() to define new variables, in Plugins, the only valid usage of Set() is where it is already explicitly enabled (such as setting the fields on the current entity).

Plugins do support With(), which can be used to create aliases and factor the expressions.

Key changes from current behavior:

1. Do not use Patch to set the current record - instead use Set(). Patching current record will trigger a new update event which will re-run the plugin and can cause an infinite loop.
2. Just like in Power Apps, tables must be explicitly specified in the plugins' description in order for the symbol to be imported. This signifcantly helps with intellisense.

Tables use the plural display name and are serialized as logical names.
If there is a conflict in display name, Power Fx will disambiguate by using Display (Logical)

Access Connectors

Plugins supports accessing connectors. This should use the same syntax as the connector expressions in Power Apps.

Access current user

Plugins run in the context of a user and can get access to that user via the standard User object. See #2318 for details on user.
Notably, this includes the DataverseUserId field which can be used to resolve the current user against the 'Users' table.

LookUp(Users, ThisRecord.User = [@User].DataverseUserId)

Instant Plugins (Unbound Actions)

Instant plugins are Unbound Actions and don't have an explicit entity.
They receive a set of input parameters and return a set of output parameters.

inputs and outputs

The input parameter are incoming symbols.

They can return a record, where each field is an output parameter.
For example, this expression will take 2 input parameters (x,y) and return 2 output parameters (add, mul):

{ add: x + y, mul : x * y }

New behavior
If the plugin defines only a single output parameter, we can streamline and the expression can just return the single value.
x+y

Types

This just supports primitive parameter types.

To pass an entity, pass the guid and lookup. For example, to return a Name parameter that lookups up Input_Name

{Name: LookUp(Accounts, ThisRecord.Account=Guid(Input_Name)).'Account Name' }

Automated Plugins

Automated plugins run with a Current Entity. They may need to access fields from the entity and see which fields are changed (in an "Update" message).

They don't have an explicit return value. but they may need to:

• update fields in the entity (default values).
• fail the transaction (validation rules)
• produce side-effects (such as writing to other tables for logging, auditing, or setting up relationships)

Automated plugins: Referring to current and previous record

There are special objects:

• OldRecord - for Update and Delete, this refers to the previous record. This is the same as LookUp(Table, ThisRecord.Id = NewRecord.Id)

• NewRecord - for Create and Update messages - this refers to the record with the updates applied. For Create pre-op, since the entity is not yet created, NewRecord's primary key id will blank. NewRecord also has all fields filled out (not just the field that changed).

• TBD: we may add CurrentRecord for other future messages such as Read.

This allows rules like:

If(NewRecord.Name <> OldRecord.Name, 
Error("Can't change name")
)

There is no explicit scope, so you must say NewRecord.Field1 and not just Field1.

Updating fields in the current entity

For Automated plugins, to update fields in the current entity, use Set().
This will update the Current Entity (and not trigger a write to the database until the plugin succeeds and the plugin infrastructure persists the change back to the database)

Set(NewRecord.Field1, NewRecord.Field2 * 10)

Set() on OldRecord is a failure since OldRecord is not mutable.

Updating fields on Read

We don't currently support Read messages, but once we do - Set becomes very important as a means to calculate on-demand fields (that are not persisted in the database). This is how formula columns work today. And it will be very useful for creating Power Fx columns on Virtual Tables (where the rows are external to the database)

Eventing

This section explain the interaction with the eventing pipeline in more detail.

1. Messages: This focuses on the Create, Update, and Delete messages.
2. Stages: This focuses on pre-op (before the operation is written to database) and post-op (after written). Both stages fall under the transaction.

Some key observations:

1. Create pre-op does not yet have NewRecord' primary key since the entity hasn't been created yet.
2. Set() is blocked in post-op. To modify current fields, use Set() in pre-op.
3. Patch() on NewRecord is blocked since this likely leads to infinite cycles and makers should be using Set instead. This will be blocked at compile time, and an extra safety check at runtime to immediately error (which is hopefully easy to diagnose) instead of allowing a write that would lead to an infinite cycle (which would be hard to diagnose).
4. LookUp/Filter/etc will always be "truthful" and directly call the plugin's IOrganization service and behave consistently with what a C# pro-dev plugin would do. For example, a LoookUp() in Update pre-op will return the record prior to updates (OldRecord), but the same operation in post-op will now return the record with updates applied.

Migration

This calls out some notable differences from the existing implementation.
Our plan to mitigate impact is:

1. Apply rewriters to update expressions. for example, the plugin infra can automatically rewrite XPrevRecord() to OldRecord in the maker's expressions. This is the same model that Power Apps follows when they make breaking changes.
2. All new plugins will default to new rules, but continue to allow old behavior to give customers a chance to migrate.

Open Questions

• should we block OldRecord in post-op? To access OldRecord, use pre-op.
• Should we block Delete Post-op. This only gets a Guid and has limited ability. Use Delete Pre-op instead.
• Overlap with Formula Columns. Formula Columns compile to SQL and run against the database (effectively as post-op operations). This can be counter-intuitive when using a formula column in pre-op. For example, if a formula column defines Field2 as Field1 * 10, then an Update Pre-op plugin will see the new value of Field1 with the old value of Field2. Can we mitigate the confusion?

[filcole]

Hi @mike, is this implemented in the current low code plugins by via the accelerator because Nathan Rose found out that Set() is not longer a valid function, and that Patch() had to be used. I have tested this too.

I've been looking at how sync classic workflows might map/convert to low code plugins. I think Error() should stop the low code plugin without any PowerFx after the Error() executing. It will be confusing to people if the following creates an annotation.

If(NewRecord.Name <> OldRecord.Name,
Error("Can't change name")
);
Patch('Notes', { Summary: "name changed"});

[MikeStall]

One challenge is consistency. If we shim Patch - and have it special-case when modifying the Current record, then we also have to Shim Filter/LookUp and all other functions to maintain that consistency.

Patch(Table, NewRecord, { field1: 100} );  
LookUp(Table, Id=NewRecord.Id).field1  // expect new value?

I'm wondering if there's a case where one of these options leads to the "wrong" thing.

[nathanroseakl]

@MikeStall sorry I'm not understanding. Are you proposing that Patch() also replace Collect() so that it works the way it does in a canvas app i.e. patch(defaults.....? If you have to make substantial changes to the other functions to maintain consistency I'd say that's probably more trouble than it's worth - I'd rather get it in people's hands sooner rather than later. Is your concern having to refactor post ga?

[filcole]

Hi @MikeStall & @nathanroseakl,

Just created a new DV environment in US Preview, with DV Accelerator Solution at version 1.0.4.36, and I'm still seeing that Set() is no longer available. So it makes it a bit hard to test.

In earlier versions of the DV accelerator, when Set() was allowed, I found it had issues with setting a lookup. In classic workflows there are often cases where Lookup values are hardcoded, and I was not able find a way to use Set() to set a lookup field to a consistent fixed value, e.g. this example from my notes

Const05 Account Set PrimaryContact to existing contact

Note the contact record must already be in the system, and since it is being referred to directly the classic workflow will have a hardcoded GUID.

Patch('Accounts', ThisRecord, { 'Primary Contact': LookUp('Contacts', 'Full Name' = "James Bond") });

Yaml:

Patch(account, ThisRecord, { primarycontactid: LookUp(contact, fullname = "James Bond") });

Note: Previously the following was not possible using the Set() functionality:
Set('Primary Contact', LookUp('Contacts', 'Full Name' = "James Bond") ); - DOES NOT WORK

Am I using Set() wrong or is that a limitation that could/should be resolved?

[filcole]

@MikeStall here some more thoughts on the doc having had a second read through of it.

Just like in Power Apps, tables must be explicitly specified in the plugins' description in order for the symbol to be imported. This signifcantly helps with intellisense.

Point 1: What does this mean? How does one explicitly specify a table within the plugins' description?

Plugins run in the context of a user and can get access to that user via the standard User object. See #2318 for details on user.

Point 2: Classic Workflows can be specified to run as another user. Generally this is useful to elevate permissions so that other records can be updated programmatically (by the elevated user) when triggered by the current (less privileged) user. Only allowing low-code plugins to run as the current user will make it harder to switch to low-code plugins from classic workflows. The plugin base class output by pac plugin init provides InitiatingUserService and PluginUserService which would be good to expose to the low-code plugins.

  PluginUserService = serviceProvider.GetOrganizationService(PluginExecutionContext.UserId); // User that the plugin is registered to run as, Could be same as current user.
  InitiatingUserService = serviceProvider.GetOrganizationService(PluginExecutionContext.InitiatingUserId); //User who's action called the plugin.

OldRecord - for Update and Delete, this refers to the previous record.

Point 3: Really like having access to the pre-image if needed. 👍

NewRecord - for Create and Update messages - this refers to the record with the updates applied.

Point 4: Not sure I like the name 'NewRecord' when used for Update messages, but I can't immediately think of a better alternative. ThisRecord is taken, CurrentRecord is mentioned for read scenarios. ContextRecord? ProcessingRecord? MyRecord? EventRecord?

Set(Field1, Field2 * 10)

Point 5: Why Set() vs Patch(), it feels like Set() should be used to set a variable, and not a table field. If the low-code plugin is still within the DB transaction then does using Patch() multiple times within the same plugin on the same table cause issues?

1. Create pre-op does not yet have NewRecord' primary key since the entity hasn't been created yet.

Point 6: What if somebody tries to use Set() to set the primary key during a create?

3. Patch() on NewRecord is blocked.

Point 7: I think I'd still want to have the ability to Patch() the triggering table, i.e. to update a related record to the newly created record in the same table. Will this be blocked?

Do we forbid implicit scope .... i.e. allow: Set(NewRecord.field1, ...)

Point 8: The Set(NewRecord.field1, ...) syntax looks ugly, much prefer Set(field1, ...) if using Set();

Overlap with Formula Columns

Point 9: Could it be determined that a formula field is being used where it's contents may be stale, i.e. by if one of the fields in the expression used for calculated column exists in the fx expression dependencies? If then could warning 'squiggles' be put underneath it's usage at design/save time?

Point 10: In the current dataverse accelerator UI the same FX expression needs to be re-entered/rekeyed when the same logic is used in both Create and Update events. In classic workflows the same XAML workflow can be triggered. It would be great if the editor could avoid that issue - it seems that this is a matter of UI though. It looks like the same fxexpression could be linked to two plugin steps (one for Create, one for Update)?

[rnwood]

I understand the distinction between Set() and Update(), but prefer the Set(TargetRecord.Field, value) suggestion over just Set(Field, value) as it's clearer than you are setting something in that record as opposed to a variable. Variables should be usable too and having them both in the same name space seems dangerous and confusing.

[magesoe]

I suggest getting rid of the NewRecord term. Only have OldRecord and CurrentRecord. This should also be available post op to check whether a value has changed.

Don't block any registration that would be possible in a plugin. So delete post op is completely valid.

Sounds like a good idea to forbid implicit scope, or force implicit scope to be CurrentRecord would be fine as well.

What is the reasoning for not having formula columns as Retrieve plugins. This would mitigate this issue

[MikeStall]

"What is the reasoning for not having formula columns as Retrieve plugins"
Are you asking why we only support Create/Update/Delete and not yet Read? We will support Read eventually - it's just a "crawl/walk/run" mentality to start with limited scope, do a great job, and then expand on that scope.
In particular, Read will be a great future option for dealing with any limitations in Power Fx formula columns.

"Don't block any registration that would be possible in a plugin. So delete post op is completely valid."
The tradeoff we're wresting with is: how easy is it for a customer to get it wrong? If there's something that's very likely to get misunderstood, it may be safer to block it for now, and then enable later with appropriate safety fences.

Is there something specific you'd want to do in Delete post-op that couldn't be done in pre-op?

[eswarpr]

My two pence/cents:
Patch Vs Set: Keeping behaviours consistent with existing PowerFx language elements and behaviours will make it easier for Devs to adapt form their Canvas development experience. Set means something completely different in Canvas authoring, so not a great replacement for what Patch means.
NewRecord/ThisRecord: Does "NewRecord" mean the record that is being updated in a Dataverse event? If so how do we compare values between the old record and updated record?

[filcole]

Point 11: After noticing the new plugin trace viewer in the latest version of the Dataverse Accelerator solution I think it might be useful to add a Trace() or Telemetry() function to automatically log to the ilogger and/or (traditional) tracelog.

[filcole]

Point 12:

When reviewing how Classic Workflow can update the same column multiple times I wondered how this might be handled with low-code plugins. e.g. The following classic workflow updates the account name of triggering record twice, once to set the account name to 'Stage 01' and then to append 'stage2' to the newly updated account name.

The result of this is an account with account name set to 'Stage 01 stage2'.

How might one mirror this within PowerFx?

Trying the naïve double patch ignores the first Patch().

Would it be possible NewRecord or CurrentRecord, perhaps similar to the below to update the name twice?

Set('Account Name', "Stage 01");
Set('Account Name', Concatenate(CurrentRecord.'Account Name', " Stage2");

I realise this is trivially re-written to Set('Account Name', "Stage 01 Stage2"); but in more complicated scenarios, especially when considering classic sync workflows this behaviour is relevant.

[dobi-sal]

Hello @MikeStall,
when should we expect Low Code Plugins to become General Available in Power Apps?

Thanks.

[AlexanderPavlovv]

I've been wondering the same, when it will no longer be a 'Preview' feature?

[KGMunaweera]

Is it possible to run automated plugin when a record is related / unrelated (associated/dissociated) between 2 tables having a relationship?

[etn-v]

anyone has success by updating a LookUp column with Set for a new record?
As previously said "Set(NewRecord.LookUpColumn, LookUp(LookUpTable, LookUp Arguments) )" is not working

[chumleesockson]

@MikeStall Are you able to help with the lookup issue mentioned above? I just published a blog post on low-code plug-ins and I'm looking to do more, but it feels like they're still not quite ready for us to use, even for playing around with.

For example, if you edit an existing plug-in, the table is no longer there and the column references are switched to the logical names instead of the display names.

All of the above is happening with the latest version of the Dataverse Accelerator app (the environment is on Dataverse version 9.2.24095.192).