Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[core] Use Instant internally for DateTimeType #3583

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

jlaur
Copy link
Contributor

@jlaur jlaur commented Apr 29, 2023

Proposal for switching DateTimeType to use Instant internally rather than ZonedDateTime.

This will effectively discard time-zone information from the source of the DateTimeType and leave it to consumers to apply time-zone at the moment of presentation, according to the configured time-zone in openHAB regional settings or in the browser.

Configured time-zone is now applied for:

  • REST API
  • SSE item state events
  • Item UI registry (affecting e.g. sitemaps)

Consequential changes because DateTimeType is now "zone-less":

The following demonstrates the difference between old and new behavior.

REST API calls are for http://localhost:8080/rest/items/DateTimeTest and the returned state is shown in the comparison table:

{
    "link": "http://localhost:8080/rest/items/DateTimeTest",
    "state": "2024-10-31T21:00:00.000+0100",
    "stateDescription": {
        "pattern": "%1$tY-%1$tm-%1$td %1$tH:%1$tM:%1$tS",
        "readOnly": false,
        "options": []
    },
    "editable": true,
    "type": "DateTime",
    "name": "DateTimeTest",
    "label": "",
    "category": "",
    "tags": [],
    "groupNames": []
}

Updates are made in Karaf using commands like:

openhab:update DateTimeTest 2024-10-31T21:00:00+01:00
Step Before Now
System time-zone: Europe/Copenhagen
Set openHAB time-zone: Europe/Copenhagen
Update to: 2024-10-01T20:00:00Z
Observe REST API result 2024-10-01T20:00:00.000+0000 2024-10-01T22:00:00.000+0200
Observe /settings/items/ 2024-10-01T20:00:00.000+0000 2024-10-01T22:00:00.000+0200
Observe /settings/items/DateTimeTest 2024-10-01 22:00:00 2024-10-01 22:00:00
Observe sitemap 2024-10-01 22:00:00 2024-10-01 22:00:00
Update to: 2024-10-31T20:00:00Z
Observe REST API result 2024-10-31T20:00:00.000+0000 2024-10-31T21:00:00.000+0100
Observe /settings/items/ 2024-10-31T20:00:00.000+0000 2024-10-31T21:00:00.000+0100
Observe /settings/items/DateTimeTest 2024-10-31 21:00:00 2024-10-31 21:00:00
Observe sitemap 2024-10-31 21:00:00 2024-10-31 21:00:00
Update to: 2024-10-31T21:00:00+01:00
Observe REST API result 2024-10-31T21:00:00.000+0100 2024-10-31T21:00:00.000+0100
Observe /settings/items/ 2024-10-31T21:00:00.000+0100 2024-10-31T21:00:00.000+0100
Observe /settings/items/DateTimeTest 2024-10-31 21:00:00 2024-10-31 21:00:00
Observe sitemap 2024-10-31 21:00:00 2024-10-31 21:00:00
Set time-zone: US/Alaska
Observe REST API result 2024-10-31T21:00:00.000+0100 2024-10-31T12:00:00.000-0800
Observe /settings/items/ 2024-10-31T21:00:00.000+0100 2024-10-31T12:00:00.000-0800
Observe /settings/items/DateTimeTest 2024-10-31 21:00:00 2024-10-31 12:00:00
Observe sitemap 2024-10-31 21:00:00 2024-10-31 12:00:00
Update to: 2024-10-31T12:00:00-0800
Observe REST API result 2024-10-31T12:00:00.000-0800 2024-10-31T12:00:00.000-0800
Observe /settings/items/ 2024-10-31T12:00:00.000-0800 2024-10-31T12:00:00.000-0800
Observe /settings/items/DateTimeTest 2024-10-31 21:00:00 2024-10-31 12:00:00
Observe sitemap 2024-10-31 21:00:00 2024-10-31 12:00:00

As it can be seen from the table, we now have consistent results.

Example DSL rule:

var dt = ZonedDateTime.of(2024, 4, 26, 0, 0, 0, 0, ZoneId.systemDefault())
DateTimeTest1.postUpdate(new DateTimeType(dt))
DateTimeTest2.postUpdate(new DateTimeType(dt.toInstant().atZone(ZoneId.of("UTC"))))
DateTimeTest3.postUpdate(new DateTimeType(dt.toInstant()))

Previous result with system time-zone CET/DST:

[openhab.event.ItemStateChangedEvent ] - Item 'DateTimeTest1' changed from NULL to 2024-04-26T00:00:00.000+0200
[openhab.event.ItemStateChangedEvent ] - Item 'DateTimeTest2' changed from NULL to 2024-04-25T22:00:00.000+0000

New result with same system time-zone:

[openhab.event.ItemStateChangedEvent ] - Item 'DateTimeTest1' changed from NULL to 2024-04-26T00:00:00.000+0200
[openhab.event.ItemStateChangedEvent ] - Item 'DateTimeTest2' changed from NULL to 2024-04-26T00:00:00.000+0200
[openhab.event.ItemStateChangedEvent ] - Item 'DateTimeTest3' changed from NULL to 2024-04-26T00:00:00.000+0200

Summary of benefits for developers:

  • In cases where TimeZoneProvider is used only to generate a DateTimeType with the correct time-zone, this can now be completely removed. This simplifies the code. See Simplify DateTimeType handling openhab-addons#17725 for examples.
  • The default DateTimeType constructor can now be used (without losing correct time-zone).
  • In cases where the configured time-zone was not correctly applied, this will now work properly without any changes.
  • The change is fully backwards compatible, although two methods have been deprecated: toZone and toLocaleZone. I will go through usages and remove them, since they no longer make sense. See Simplify DateTimeType handling openhab-addons#17725.
  • Pull request reviews should be a bit smoother, since add-on maintainers will no longer have to assist contributors in constructing a proper DateTimeType using TimeZoneProvider.

And for users:

  • There will now be consistent behavior across all bindings, where previously time-zone could differ between bindings and even within same binding: System time-zone or openHAB time-zone.
  • There will now be consistent behavior across different parts of the UI, where previously some parts used the time-zone contained in the DateTimeType (which could be either depending on binding) and other parts used system time-zone.
  • A change in time-zone will have immediate effect for the UI without requiring state updates for the items.

And finally limitations:

  • It is no longer possible to send a DateTimeType command to a binding with a specified time-zone. The bindings will have to provide a time-zone if needed (e.g. time-zone from device, openHAB configuration or system time-zone). I have not found any bindings using this possibility, so I don't think this will cause any issues.

Resolves #2898

@jlaur jlaur force-pushed the 2898-datetimetype-instant branch 2 times, most recently from ab14428 to 81c3dc0 Compare April 29, 2023 22:36
@jlaur
Copy link
Contributor Author

jlaur commented Sep 19, 2023

I hit a wall with this when realizing that toFullString() and toString() should probably format as UTC for consistency. However, I believe one of these is used by Main UI for display purposes, so to improve the display with correct time-zone rather breaking it by raw UTC format, we would need UI changes - see also #2898 (comment).

I don't know if any @openhab/webui-maintainers can throw some light at what may be a possible way forward?

@jlaur jlaur force-pushed the 2898-datetimetype-instant branch 2 times, most recently from 4ec7437 to 229ee4e Compare December 20, 2023 14:28
@wborn wborn added the work in progress A PR that is not yet ready to be merged label Jan 4, 2024
@jlaur jlaur force-pushed the 2898-datetimetype-instant branch from 229ee4e to 266b055 Compare April 10, 2024 18:47
@jlaur
Copy link
Contributor Author

jlaur commented Apr 10, 2024

@mherwege - seeing your work in #4169 I'm wondering if you can advise how to accomplish decoupling the time-zone + formatting of DateTimeType. The general idea is to reduce it to be only an Instant without time-zone information. Therefore the UI should no longer display the raw output of toString() / toFullString(). Instead it must use the configured time-zone from TimeZoneProvider for formatting the DateTime in local time.

Currently it's formatted like this: 2024-04-10T20:55:01.000+0200. The problem is that "+0200" is the time-zone provided when updating the value, not the current time-zone. I don't know how this formatting works in Main UI.

@mherwege
Copy link
Contributor

mherwege commented Apr 11, 2024

@jlaur Main UI works with what it gets from the REST API.
On the Items screen (items list), it will show the state exactly as it is received from the REST API /items endpoint.
Here is an extract of such a response:

    {
        "link": "http://openhabian.local:8080/rest/items/PMD",
        "state": "2024-04-11T00:00:00.000+0200",
        "stateDescription": {
            "pattern": "%1$td-%1$tm-%1$tY",
            "readOnly": true,
            "options": []
        },
        "editable": true,
        "type": "DateTime",
        "name": "PMD",
        "label": "PMD",
        "category": "calendar",
        "tags": [],
        "groupNames": [
            "Afval"
        ]
    }

state is what gets shown in the items list.

The formatted state comes from the states event stream in the displayState field. It is a result of applying the stateDescription pattern, but that is done in core.
Event stream extract:

data: {"Afval":{"state":"2024-04-10T00:00:00.000+0200","displayState":"10-04-2024","type":"DateTime"},"Papier":{"state":"2024-04-10T00:00:00.000+0200","displayState":"10-04-2024","type":"DateTime"},"Restafval":{"state":"2024-04-18T00:00:00.000+0200","displayState":"18-04-2024","type":"DateTime"},"PMD":{"state":"2024-04-11T00:00:00.000+0200","displayState":"11-04-2024","type":"DateTime"}}

displayState gets used in widgets when available.

So, to understand where state and displayState come from, one needs to look at the REST API code. According to

private Collection<Item> getItems(@Nullable String type, @Nullable String tags) {

state is added through the EnrichedItemDTOMapper here:
So it is simply using the toFullString() method on the state of the item.

@jlaur
Copy link
Contributor Author

jlaur commented Apr 11, 2024

@mherwege - many thanks for your insights. I will have a closer look at this soon.

@openhab-bot
Copy link
Collaborator

This pull request has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/use-instant-in-historicitem-gettimestamp-instead-of-zoneddatetime/158302/3

@jlaur jlaur force-pushed the 2898-datetimetype-instant branch 7 times, most recently from 0829630 to d007aa1 Compare November 7, 2024 21:06
@jlaur jlaur marked this pull request as ready for review November 8, 2024 07:31
@jlaur jlaur requested a review from a team as a code owner November 8, 2024 07:31
@jlaur
Copy link
Contributor Author

jlaur commented Nov 8, 2024

@openhab/core-maintainers - I don't have permission to remove the "work in progress" label since I'm not in the contributors team, so perhaps someone can do that for me? I consider this PR ready for review now after finally fixing the returned state in the REST API. In parallel I will run some more tests and also see if it is possible to use TimeZoneProvider in more places rather than converting to system time-zone.

@mherwege - perhaps you'd also like to have a look?

@lolodomo
Copy link
Contributor

lolodomo commented Nov 8, 2024

What about bindings setting a DateTimeType state with a particular time zone?
Is it really backward compatible with all bindings?

@jlaur
Copy link
Contributor Author

jlaur commented Nov 8, 2024

What about bindings setting a DateTimeType state with a particular time zone?

They can still do that, but the time-zone will be stripped, so it's really not needed anymore.

Is it really backward compatible with all bindings?

It should be, but there can be unit tests failing if they expect a certain time-zone:

DateTimeType dt = new DateTimeType(ZonedDateTime.now());
ZonedDateTime zdt = dt.getZonedDateTime();

Previously this would return a ZonedDateTime with the same zone as provided when creating the object. But since the zone is now stripped, it will return a ZonedDateTime with system time-zone. Generally getInstant() should be preferred over getZonedDateTime() now.

@jlaur
Copy link
Contributor Author

jlaur commented Nov 8, 2024

@joerg1985 - perhaps you would also like to have a look? I saw some of your recent related comments and enhancements, e.g. #4384.

@jlaur
Copy link
Contributor Author

jlaur commented Nov 8, 2024

This will call DateTimeType.toFullString:

ItemEventPayloadBean bean = new ItemEventPayloadBean(getStateType(state), state.toFullString());

which is probably the reason for system time-zone being used in some places. I don't know how to improve this, since TimeZoneProvider is required for properly formatting this in the configured local time-zone. One solution might be to change the implementation of DateTimeType.toFullString to format in UTC as Instant. However, this would require all receivers to reformat by applying local time-zone from TimeZoneProvider, so I think it would be a bigger change. Any suggestions will be much appreciated here.

Keeping it "as is", in current version of the code I have provided, will be correct in all cases where system time-zone is the same as the time-zone configured in openHAB. When they are different and a state is provided using the correct time-zone from TimeZoneProvider (i.e. from bindings correctly implementing this), it might have been correctly displayed in local time-zone previously, but will now be in system time-zone. However, I have not been able to reproduce this. As it can be seen in the PR description table, even providing a different time-zone when updating items, resulted in system time-zone being used for formatting in UI. So perhaps "as is" keeps status quo.

@jlaur jlaur force-pushed the 2898-datetimetype-instant branch 2 times, most recently from 8a820be to 7df4b06 Compare November 8, 2024 22:47
@jlaur
Copy link
Contributor Author

jlaur commented Nov 11, 2024

@wborn - can you help me by removing the "work in progress" label to signal this is now ready for the review process? 🙂

@holgerfriedrich holgerfriedrich removed the work in progress A PR that is not yet ready to be merged label Nov 11, 2024
@@ -51,6 +56,10 @@ public class EnrichedItemDTOMapper {

private static final Pattern EXTRACT_TRANSFORM_FUNCTION_PATTERN = Pattern.compile("(.*?)\\((.*)\\):(.*)");

private static final String DATE_FORMAT_PATTERN_WITH_TZ_RFC = "yyyy-MM-dd'T'HH:mm[:ss[.SSSSSSSSS]]Z";
private static final DateTimeFormatter FORMATTER_TZ_RFC = DateTimeFormatter
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this pattern identical to DateTimeFormatter.ISO_OFFSET_DATE_TIME? In any case the .SSSSSSSSS is incorrect, should be .nnnnnnnnn in my mind.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have to admit this was duplicated from DateTimeType itself, thinking I would later remove it there - which didn't happen. I'll see if it can be refactored while maintaining backwards compatiblity, and also get rid of the duplication.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In DateTimeType I have now exposed a method overload of toFullString taking a ZoneId - and added full test coverage for this. As a consequence, this duplicated method and DateTimeFormatter was removed.

For now, I have left the method and DateTimeFormatter as it was before this PR, but we now have test coverage to make sure refactoring will not break the backwards compatibility.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems DateTimeFormatter.ISO_OFFSET_DATE_TIME does not provide the same output:

[ERROR] Failures: 
[ERROR]   DateTimeTypeTest.createDate:360 expected: <2014-03-30T10:58:47.033+0000> but was: <2014-03-30T10:58:47.033Z>
[ERROR]   DateTimeTypeTest.createDate:360 expected: <2014-03-30T12:58:47.033+0200> but was: <2014-03-30T12:58:47.033+02:00>
[ERROR]   DateTimeTypeTest.createDate:360 expected: <2014-03-30T12:58:47.033+0200> but was: <2014-03-30T12:58:47.033+02:00>
[ERROR]   DateTimeTypeTest.createDate:360 expected: <2014-03-30T05:58:47.033-0200> but was: <2014-03-30T05:58:47.033-02:00>
[ERROR]   DateTimeTypeTest.createDate:360 expected: <2014-03-30T12:58:47.033-0200> but was: <2014-03-30T12:58:47.033-02:00>
[ERROR]   DateTimeTypeTest.createDate:360 expected: <1970-01-01T10:58:47.000+0000> but was: <1970-01-01T10:58:47Z>
[ERROR]   DateTimeTypeTest.createDate:360 expected: <1970-01-01T10:58:00.000+0000> but was: <1970-01-01T10:58:00Z>
[ERROR]   DateTimeTypeTest.createDate:360 expected: <1970-01-01T10:58:47.000+0100> but was: <1970-01-01T10:58:47+01:00>
[ERROR]   DateTimeTypeTest.createDate:360 expected: <1970-01-01T10:58:00.000+0100> but was: <1970-01-01T10:58:00+01:00>
[ERROR]   DateTimeTypeTest.createDate:360 expected: <2014-03-30T00:00:00.000+0000> but was: <2014-03-30T00:00:00Z>
[ERROR]   DateTimeTypeTest.createDate:360 expected: <2014-03-30T10:58:47.033+0000> but was: <2014-03-30T10:58:47.033Z>
[ERROR]   DateTimeTypeTest.createDate:360 expected: <2014-03-30T08:58:47.033+0000> but was: <2014-03-30T08:58:47.033Z>
[ERROR]   DateTimeTypeTest.createDate:360 expected: <2014-03-30T10:58:47.230+0000> but was: <2014-03-30T10:58:47.23Z>
[ERROR]   DateTimeTypeTest.createDate:360 expected: <2014-03-30T10:58:47.000+0000> but was: <2014-03-30T10:58:47Z>
[ERROR]   DateTimeTypeTest.createDate:360 expected: <2014-03-30T12:58:47.033+0200> but was: <2014-03-30T12:58:47.033+02:00>
[ERROR]   DateTimeTypeTest.createDate:360 expected: <2014-03-30T10:58:47.033+0200> but was: <2014-03-30T10:58:47.033+02:00>
[ERROR]   DateTimeTypeTest.createDate:360 expected: <2014-03-30T10:58:47.000+0200> but was: <2014-03-30T10:58:47+02:00>
[ERROR]   DateTimeTypeTest.createDate:360 expected: <2014-03-30T15:58:47.000+0500> but was: <2014-03-30T15:58:47+05:00>
[ERROR]   DateTimeTypeTest.createDate:360 expected: <2014-03-30T10:58:47.000+0200> but was: <2014-03-30T10:58:47+02:00>
[ERROR]   DateTimeTypeTest.toFullStringWithZone:412 
Expected: is "2024-11-11T20:39:00.000+0000"
     but: was "2024-11-11T20:39:00Z"
[ERROR]   DateTimeTypeTest.toFullStringWithZone:412 
Expected: is "2024-11-11T20:39:00.000+0000"
     but: was "2024-11-11T20:39:00Z"
[ERROR]   DateTimeTypeTest.toFullStringWithZone:412 
Expected: is "2024-11-11T20:39:00.000000001+0000"
     but: was "2024-11-11T20:39:00.000000001Z"
[ERROR]   DateTimeTypeTest.toFullStringWithZone:412 
Expected: is "2024-11-11T20:39:00.123+0000"
     but: was "2024-11-11T20:39:00.123Z"
[ERROR]   DateTimeTypeTest.toFullStringWithZone:412 
Expected: is "2024-11-11T20:39:00.123456+0000"
     but: was "2024-11-11T20:39:00.123456Z"
[ERROR]   DateTimeTypeTest.toFullStringWithZone:412 
Expected: is "2024-11-11T20:39:00.123456789+0000"
     but: was "2024-11-11T20:39:00.123456789Z"
[ERROR]   DateTimeTypeTest.toFullStringWithZone:412 
Expected: is "2024-11-11T21:39:00.123+0100"
     but: was "2024-11-11T21:39:00.123+01:00"
[ERROR]   DateTimeTypeTest.toFullStringWithZone:412 
Expected: is "2024-11-10T23:59:59.999-0500"
     but: was "2024-11-10T23:59:59.999-05:00"

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The parser test does accept the format above as valid input, so why should this be invalid output?

Copy link
Contributor Author

@jlaur jlaur Nov 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With da2b50a this duplicated method was removed, so what is left is the original implementation of toString()/toFullString(). This is unchanged by this PR to remain backwards compatible. The DateTimeType String constructor supports many different inputs, but toFullString() can only provide one specific output. Other outputs are supported by the format methods.

if (!sign.isEmpty()) {
// the formatted string contains 9 fraction-of-second digits
// truncate at most 2 trailing groups of 000s
return formatted.replace("000" + sign, sign).replace("000" + sign, sign);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the formater mentioned above this should not be needed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jlaur
Copy link
Contributor Author

jlaur commented Nov 12, 2024

which is probably the reason for system time-zone being used in some places. I don't know how to improve this, since TimeZoneProvider is required for properly formatting this in the configured local time-zone. One solution might be to change the implementation of DateTimeType.toFullString to format in UTC as Instant. However, this would require all receivers to reformat by applying local time-zone from TimeZoneProvider, so I think it would be a bigger change. Any suggestions will be much appreciated here.

This was not the case. I finally figured it out and fixed this inconsistent behavior, so that configured time-zone rather than system time-zone is now used for SSE item events and sitemaps as well.

jlaur added a commit to jlaur/openhab-docs that referenced this pull request Nov 13, 2024
jlaur added a commit to jlaur/openhab-docs that referenced this pull request Nov 13, 2024
@jlaur jlaur force-pushed the 2898-datetimetype-instant branch 2 times, most recently from d691ee3 to be50c88 Compare November 14, 2024 18:56
@jlaur
Copy link
Contributor Author

jlaur commented Nov 18, 2024

Is it really backward compatible with all bindings?

It should be, but there can be unit tests failing if they expect a certain time-zone:

DateTimeType dt = new DateTimeType(ZonedDateTime.now());
ZonedDateTime zdt = dt.getZonedDateTime();

Previously this would return a ZonedDateTime with the same zone as provided when creating the object. But since the zone is now stripped, it will return a ZonedDateTime with system time-zone. Generally getInstant() should be preferred over getZonedDateTime() now.

I have now checked all bindings by recompiling main against changes of this PR and fixed those tests having that assumption: openhab/openhab-addons#17764.

@mherwege
Copy link
Contributor

@jlaur I looked at the code, and there is nothing I can add to this. I agree this will simplify addon development and make things more consistent.
The one worry I have is that we are getting close to the 4.3 release. I am not sure we will have enough milage in a milestone for this. It doesn't look like there is much impact, but touching a core type always carries some risk. But I let maintainers be the judge on this.

From my perspective LGTM.

@jlaur
Copy link
Contributor Author

jlaur commented Nov 24, 2024

@openhab/core-maintainers - WDYT?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[DateTimeType] ZonedDateTime, Instant or neither?
7 participants