enable input of effectiveDate for refreshes

[samaguire]

You may also want to consider allowing the user to input the effectiveDate. If ommitted it defaults to UTC, which for me being UTC+12/13 in NZ can cause issues when incrementing by day. I use the format "yyyy-MM-ddTHH:mm:ss.fffffffZ".

see previous post: #27 (comment)

for example

[gbrueckl]

on a dataset-level or on a table level?

i could probably overwrite the UTC time (API default) with your current local time - would that help?
whats the default in the Power BI UI when I click refresh on a DAtaset- it will do the incremental refresh only, right?

I am still not sure if it should be integrated into the UI or just available as API call via notebooks (as it is of today)
btw. you can also select multiple partitions or tables and drag&drop them into your notebook

[samaguire]

This is a complicated topic.

To keep it simple, I had thought that you could have a popup input box which populated by default with the local date time formatted as "yyyy-MM-ddTHH:mm:ss.fffffffZ" after selecting the refresh type. The input could be controlled by a parameter, so that the user can optionally control if it is specified and send as part of the request.

I was specifically thinking of a refreshType of full or automatic against a table with an incremental refresh policy defined to increment at the day level. Anything beyond this, i.e. doing the refresh as a commitMode of partialBatch would be an advanced refresh.

I've attached my notebook which I've refined to the sort of use cases I use (Async Refresh.zip). I agree that something beyond just triggering a simple refresh should go through notebooks. (I also didn't know about the drag and drop of tables and partitions, very handy!)

Triggering a refresh in the service executes the equivalent of the below Enhance API call, which creates new incremental partitions, updates historical incremental partitions when in scope, and all other partitions enabled for refresh without an incremental policy. A key difference though, is that the service calculates the effectiveDate in the time zone of the refresh schedule set for the model, while the Enhanced API doesn't (https://learn.microsoft.com/en-us/power-bi/connect-data/incremental-refresh-overview#current-date-and-time).

{
    "type": "full",
    "commitMode": "transactional",
    "applyRefreshPolicy": "true",
    "effectiveDate": "$(EFFECTIVEDATE)"
}

[gbrueckl]

would you do the incremental refresh on a single/set of tables or on the whole dataset?
until I have investigated the table/partitions of a dataset, I do not know whether Incremental Refresh is enabled or not.

What I am currently thinking of is that next to all the other Refresh-Types (Full, Calculated, Clear,...) I would also provide the option "Incremental Refresh", which would then ask you for the effectiveDate as you already proposed

but I would only like to display this option if Incremental Refresh is actually enabled on the Dataset - so I would like to figure this out first

[samaguire]

I'd potentially do both. Testing I'd do specific tables, but going to Prod I'd want to do the whole lot. Yeah, there would be a bit of work to determine if an incremental policy was set. I do like the idea of creating incremental versions for each of the types. 👍 Regards Stephen Get Outlook for Android<https://aka.ms/ghei36>

…

________________________________ From: Gerhard Brueckl ***@***.***> Sent: Thursday, April 11, 2024 6:36:03 PM To: gbrueckl/PowerBI-VSCode ***@***.***> Cc: Stephen Maguire ***@***.***>; Author ***@***.***> Subject: Re: [gbrueckl/PowerBI-VSCode] enable input of effectiveDate for refreshes (Issue #28) would you do the incremental refresh on a single/set of tables or on the whole dataset? until I have investigated the table/partitions of a dataset, I do not know whether Incremental Refresh is enabled or not. What I am currently thinking of is that next to all the other Refresh-Types (Full, Calculated, Clear,...) I would also provide the option "Incremental Refresh", which would then ask you for the effectiveDate as you already proposed but I would only like to display this option if Incremental Refresh is actually enabled on the Dataset - so I would like to figure this out first — Reply to this email directly, view it on GitHub<#28 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AO3PBAXQSH5NB2Q72SD4TETY4YVNHAVCNFSM6AAAAABFW4JS3SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBZGAYTINRTGI>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[gbrueckl]

ok, I can get the Refresh policy information from here https://dax.guide/info-refreshpolicies/
will try to implement this in the next days

[samaguire]

Awesome! Regards Stephen Get Outlook for Android<https://aka.ms/ghei36>

…

________________________________ From: Gerhard Brueckl ***@***.***> Sent: Thursday, April 11, 2024 7:28:23 PM To: gbrueckl/PowerBI-VSCode ***@***.***> Cc: Stephen Maguire ***@***.***>; Author ***@***.***> Subject: Re: [gbrueckl/PowerBI-VSCode] enable input of effectiveDate for refreshes (Issue #28) ok, I can get the Refresh policy information from here https://dax.guide/info-refreshpolicies/ will try to implement this in the next days — Reply to this email directly, view it on GitHub<#28 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AO3PBAU7OY4GU6EWPSJV23DY4Y3RPAVCNFSM6AAAAABFW4JS3SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBZGA4TANJUGM>. You are receiving this because you authored the thread.Message ID: ***@***.***>