v2.24.0 - Default Calendar - issue moving allday event

[evroom]

See this topic for details:

https://forum.magicmirror.builders/topic/17318/calendar-module-problem-with-moved-allday-event-google-calendar?_=1688317001295

I could apply the workaround till v2.23.0, but not anymore on v2.24.0.

admin@MagicPi3:~ $ grep version /home/admin/MagicMirror/package.json
	"version": "2.24.0",

admin@MagicPi3:~ $ lsb_release -a
No LSB modules are available.
Distributor ID:	Raspbian
Description:	Raspbian GNU/Linux 10 (buster)
Release:	10
Codename:	buster

Using: https://calendar.google.com/

Create events:

TestCal: AllDayRepeatWeekly	--> Google Calendar: Monday 3
				--> MM Default Calendar: Monday 3

TestCal: OneDayRepeatWeekly	--> Google Calendar: Tuesday 4 - 04:00 - 05:00
				--> MM Default Calendar: Tuesday 4 - 04:00 - 05:00

Note: other recurring events (AllDayRepeatWeekly and OneDayRepeatWeekly) also on MM Default Calendar

Move events two days - 'This event':

TestCal: AllDayRepeatWeekly	--> Google Calendar: Wednesday 5
				--> MM Default Calendar: Monday 3

TestCal: OneDayRepeatWeekly	--> Google Calendar: Thursday 6 - 04:00 - 05:00
				--> MM Default Calendar: Thursday 6 - 04:00 - 05:00

Note: other recurring events (AllDayRepeatWeekly and OneDayRepeatWeekly) untouched on MM Default Calendar

Delete Recurring events - 'This event':

TestCal: AllDayRepeatWeekly	--> Google Calendar: (deleted)
				--> MM Default Calendar: Monday 3

TestCal: OneDayRepeatWeekly	--> Google Calendar: (deleted)
				--> MM Default Calendar: (deleted)

Note: other recurring events (AllDayRepeatWeekly and OneDayRepeatWeekly) untouched on MM Default Calendar

Delete Recurring events - 'All events':

TestCal: AllDayRepeatWeekly	--> Google Calendar: (deleted)
				--> MM Default Calendar: (deleted)

TestCal: OneDayRepeatWeekly	--> Google Calendar: (deleted)
				--> MM Default Calendar: (deleted)

Note: other recurring events (AllDayRepeatWeekly and OneDayRepeatWeekly) deleted on MM Default Calendar

Get events:

$ rm basic.ics; wget https://calendar.google.com/calendar/ical/xxxxxxxxx/basic.ics

--> basic.ics

Collected:

basic_AllDayRepeatWeekly.ics
basic_OneDayRepeatWeekly.ics
basic_MoveEvents.ics
basic_DeleteEvents.ics
basic_DeleteAllEvents.ics

basic_DeleteAllEvents.ics.txt
basic_DeleteEvents.ics.txt
basic_MoveEvents.ics.txt
basic_OneDayRepeatWeekly.ics.txt
basic_AllDayRepeatWeekly.ics.txt

Search an event and print complete record(s):

$ awk 'BEGIN { RS="END:VEVENT" ; ORS = "END:VEVENT\n" }/AllDayRepeatWeekly/' basic.ics

[khassel]

may @sdetweil can look into this because he answered in the mentioned forum thread and opened an issue in jens-maus/node-ical#234

[sdetweil]

yes. I haven't had much time to find a solution. the rrule/luxon libs are returning incorrect dates

seems related to having byday in the rule (only specific days of the week)

[evroom]

Thanks fro the quick replies.
At the time, Sam provided me with 2 updated files:
ical.js and calendarutils.js
But the calendarutils.js file has been completely rewritten.
As a result the update in ical.js alone will not do the trick.

[khassel]

@sdetweil is this fixed with the new node-ical version?

[sdetweil]

no.. not yet fixed

[sdetweil]

see this.. waiting on user test for submitting PR here and node-ical
all testcases in both run successfully..

here is a test version of the fixes for all kinds of date problems.

best to make a new folder and git clone there

git  clone https://github.com/sdetweil/MagicMirror
cd ~/MagicMirror
git checkout fixcaldates
npm run install-mm

copy your config.js and custom.css from the prior folder
and the non-default modules you have installed..

this ONLY changes the default calendar
but DOES ship an updated node-ical library too

if you need to fall back, just rename the folders around again so that
your original is called MagicMirror

all the testcases for node-ical and MagicMirror execute successfully.

the 'BIG' change here is to get the local NON-TZ dates for the
rrule.between()

all the checking and conversion code is commented out or not used
the node-ical fixes are for excluded dates (exdate) values being adjusted for DST/STD time.. waiting to submit that PR

one fix in calendar.js for checking if a past date was too far back,
but it never checked to see IF the event date was in the past.. (before today) so it chopped off too many

and one change in calendarfetcher.js to put out a better diagnostic message of the parsed data.. (exdate was excluded cause JSON stringify couldn't convert the complex structure)

[evroom]

Testing fixes in default/calendar/ and node_modules/node-ical/ files.

Add TestCal_AllDayRepeatingEvent
Friday, October 11
All day
Weekly on Friday
5 Occurences
Result: 5 events on the list

Move first event from Friday to Saturday ('This event'):
Result: first event still on Friday, not on Saturday

Delete first event ('This event'):
Result: first event is still on the list

Delete the second event ('This event'):
Result: all 5 events still on the list

Delete the remainder of events ('All events')
Result: all events removed from the list

Same tests with a single day repeating event, with begin and end time, passed.

[evroom]

Here some data I gathered:

MMM Calendar - fixcaldates - 20241009.txt

[sdetweil]

there will be a different ics generated for each of these test cases w moved or deleted entries.
the UUID of each such modified MUST MATCH
if tye initial date changes (start) then the old one would would have an exdate entry and a new entry would be created for the new single date event on the new day

i am on my phone currently, so cant tell if these ics are correct

[evroom]

Yes, sorry, I left the command out that retrieves the latest basic.ics file.

So, after every change I did:

First:
admin@BlackPi4b:~ $ rm -f basic.ics; wget https://calendar.google.com/calendar/ical/blah-blah-blah%40group.calendar.google.com/private-blah-blah-blah/basic.ics

Then:

admin@BlackPi4b:~ $ awk 'BEGIN { RS="END:VEVENT" ; ORS = "END:VEVENT\n" }/TestCal/' basic.ics

[sdetweil]

can you provide those specific ics files, cant tell if you did

doesn't matter what it was before unless u are reporting it to the cal provider. we just see what the provider gives. (and node-ical does all the combining, so we never know directly)

[evroom]

Did not save the ics file for every part of the test, would need to re-test.
Need a moment for that.
But included in the attachment above is the
BEGIN:VEVENT / END:VEVENT
information for the related events for every test.

[sdetweil]

but your grep/awk would only see one event and there would be multiple in most cases

[sdetweil]

when u move the start, the initial event will have an exdate entry for the first, with the rrule and a second vevent for the single day with the same uid on the new day

[evroom]

My grep does show more events, when there are more present.
But in this case, because of the RRULE, it only shows one:
RRULE:FREQ=WEEKLY;WKST=MO;COUNT=5;BYDAY=FR

But in order not to discuss this and to avoid the grep not to be right I will attach the actual ics files. :-)
ics_files.tar.gz

[sdetweil]

great thx, should only show one rrule. extras don't have it, just the matching uid

[sdetweil]

hm.. I just tested 2-5 without issue.. are you sure you switched to the fixcaldates branch?

I have just added some additional formatting and debug for the issues your testcases present..
the parsed ics data was as expected and we handled it correctly.

go to the MagicMirror folder, with my branch selected (otherwise you have the broken 2.29 code)

git pull 

to update

git branch to check current branch

git branch 

.... my system output

  develop
* fixcaldates
  master

if not on my branch do

git checkout fixcaldates
npm run install-mm

output of testcases
3(move 1st) 11th->12th

4(delete 1st) no 11th

5(delate all) none

[evroom]

Sam,

I am very sorry, but I get this error when an event comes in.
When I use the previous calendarfetcherutils.js I do not get an error.

0|MagicMirror  | [2024-10-10 18:13:15.014] [DEBUG] title: TestCal - Test Single Event
0|MagicMirror  | [2024-10-10 18:13:15.019] [ERROR] Calendar Error. Could not fetch calendar:  https://calendar.google.com/calendar/ical/123456789%40group.calendar.google.com/private-123456789/basic.ics ReferenceError: d2 is not defined
0|MagicMirror  |     at /home/admin/MagicMirror/modules/default/calendar/calendarfetcherutils.js:478:24
0|MagicMirror  |     at Array.forEach (<anonymous>)
0|MagicMirror  |     at Object.filterEvents (/home/admin/MagicMirror/modules/default/calendar/calendarfetcherutils.js:145:24)
0|MagicMirror  |     at /home/admin/MagicMirror/modules/default/calendar/calendarfetcher.js:61:36
0|MagicMirror  |     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

Should be the latest files from your fixcaldates branch:

admin@BlackPi4b:~/MagicMirror $ ls -als /home/admin/MagicMirror/modules/default/calendar/calendarfetcherutils.js
24 -rw-r--r-- 1 admin admin 23166 Oct 10 17:35 /home/admin/MagicMirror/modules/default/calendar/calendarfetcherutils.js
admin@BlackPi4b:~/MagicMirror $ cksum /home/admin/MagicMirror/modules/default/calendar/calendarfetcherutils.js
3368003873 23166 /home/admin/MagicMirror/modules/default/calendar/calendarfetcherutils.js

admin@BlackPi4b:~/MagicMirror $ ls -als /home/admin/MagicMirror/modules/default/calendar/calendarfetcher.js
4 -rw-r--r-- 1 admin admin 3924 Oct 10 17:35 /home/admin/MagicMirror/modules/default/calendar/calendarfetcher.js
admin@BlackPi4b:~/MagicMirror $ cksum /home/admin/MagicMirror/modules/default/calendar/calendarfetcher.js
1727397175 3924 /home/admin/MagicMirror/modules/default/calendar/calendarfetcher.js

[sdetweil]

thanks.. git pull again

[evroom]

Sam, the new file got rid of the error I saw, but for some reason it still does not work for me.
Can it be the timezone ?
Mine being GMT+2 and yours GMT-5 ?
See the attached file for the log.

What needs to be done to have everything running in a different timezone?
Change system timezone of my Raspberry Pi using timedatectl ?
Setting in MagicMirror ?
Manual feeding calendar using a modified ics file ?

I already lived with this behaviour for a long time and refrain from using repeating all day events, so only spend time on it if you 'enjoy' it.
I would like to help, but I feel the code is rather complicated, with many dependencies and exceptions.

Greetings,

E.J.
MagicMirror.log.202410102036.txt

[sdetweil]

oh such fun, timezone are indeed a challenge

its complicated, yes, but i like the challenge, and it needs to be done. so hang in there with me

answer to your what do we need to do, as a user nothing. set your system to your timezone and all should work. you shouldn't have to think about it

[evroom]

Always happy to assist you with testing.

Last thing for today:

V - When I add a single event on 2024.10.22 from 23:00 to 00:00 then it displays correct on Oct 22nd.
Moving to Oct 26th has no issue.

X - When I add a repeating event on 2024.10.22 from 23:00 to 00:00 (repeat every 1 day - 3 occurrences) then it wrongly displays Oct 23rd, 24th and 25th (not from Oct 22nd onwards).

X - When I add a repeating event on 2024.10.22 from 22:00 to 23:30 (repeat every 1 day - 3 occurrences) then it also wrongly displays Oct 23rd, 24th and 25th (not from Oct 22nd onwards).

V - When I add a repeating event on 2024.10.22 from 21:00 to 22:00 (repeat every 1 day - 3 occurrences) then it correctly displays Oct 22nd, 23rd and 24th.
Moving the 1st event from Oct 22nd to Oct 26th has no issues.

V - When I add a repeating event on 2024.10.22 from 21:05 to 22:05 (repeat every 1 day - 3 occurrences) then it correctly displays Oct 22nd, 23rd and 24th.
Moving the 1st event from Oct 22nd to Oct 26th has no issues.

Quick tests, no logs saved.

So, it means that when you are adding repeating events within distance of GMT+0 (start time on or after GMT+0 ??) you are bound to have issues.
Non-repeating events do not have those issues.

[sdetweil]

ah, full dates were returned in UTC time instead of required local time

do

cd ~/MagicMirror
git pull
rm -rf node_modules
npm run install-mm 

to get the latest node-ical too
works correctly in your and my tz

[sdetweil]

didn't see your other info .. will check

[sdetweil]

on the kk vs HH

HH is hour 0-23
kk is hour 1-24

[evroom]

Right. And I found this explanation:

The two formats essentially do the same thing but differ in how they handle midnight.
kk will format midnight to 24:00 whereas HH will format to 00:00. 
The hours in a day in k are 1-24 and in H are 0-23

To me that explains why I was seeing 24:00.
Only a bit strange that it occurred only after moving the event and not already when adding the event.
Better to refrain form using k and/or kk altogether.

[sdetweil]

also, can you clarify what those times are ? 00:00-01:00 UTC? or? 11-12p berlin time

my calendar (google) will not allow me to start an event at midnight TODAY, (trying oct 21)
but WILL allow me to start at 00:00 the following day (oct 22) (but now the ICS says 10-22-24T00:00:00 start of day ,, not 10-21-24 , end of day)

[sdetweil]

yes, kk means to count in human terms.. 1 is the first hour of the day, not 0
I can't imagine where that would be useful.. other than in conmversation, not digital representation

[evroom]

00:00 - 01:00 Germany/Berlin time.

timeFormat: "relative",

timeFormat: "absolute",

( just updated config.js )

BEGIN:VEVENT
DTSTART;TZID=Europe/Berlin:20241021T000000
DTEND;TZID=Europe/Berlin:20241021T010000
RRULE:FREQ=DAILY;COUNT=3
DTSTAMP:20241018T180504Z
UID:[email protected]
CREATED:20241018T180449Z
LAST-MODIFIED:20241018T180449Z
SEQUENCE:0
STATUS:CONFIRMED
SUMMARY:DailyRepeatingEvent
TRANSP:OPAQUE
END:VEVENT

[evroom]

00:01 - 01:00 works fine (timeFormat: "absolute")

Moved:

[sdetweil]

that last image is clipped on the right..

I can't tell if you still have a failing testcase..

[sdetweil]

I redid a cal with berlin, 23:00-00:00 and moved 1st to last , looks good to me, added showEnd: true

[sdetweil]

note.. if you are VIEWING in a differnt timezone, then the dates/times will be changed as appropriate (I am 7 hours behind you)
viewing in US central timezone.. so we can be on the SAME event..

this looks like I am off an entire day

should be 22/23/24 at 16:00, cause 21 moved after 23

[sdetweil]

I fixed and added official testcases for moved date roll over into next day. (berlin event, one of 3 moved, viewed from berlin, chicago and sydney)

give it a try..

[evroom]

With timeFormat: "absolute" still an issue:

Git branch is fixcaldates2 and I did a git pull that pulled the changes.
Later even did a npm run install-mm, although I do not know if this was necessary.
Test results remained the same however.

Question: when I use timeFormat: "relative", should showEnd: true work ?
I only get the begin time, not the end time.
Even when setting the new, undocumented variable showEndsOnlyWithDuration: true
Can open an Issue for that, unless it is stated somewhere that relative overrules the showEnd setting.

And an other question:
Could you tell me how to use those tests under tests/configs/modules/calendar?
Perhaps it simplifies my testing, or perhaps it will show differences to what your are seeing.
I know of debug.js and const url = "http://localhost:8080/modules/calendar_tests/basic.ics";, which is also already helpful.

RepeatingEvent.Oct21.ics.txt

[sdetweil]

showEnd only works w timeFormat:"absolute"

23:59

i subtracted a second, so this happened

i don't know how to use the tests under tests except during testing (npm run test:electron) as the tests folder isnt a valid source except under test

good feedback

[sdetweil]

testcases are executed by a jest script
there are 2 main test runners

using electron, and checking content put in the web page
and end to end, which does not use electron

for electron
the calendar tests are driven w

[sdetweil]

if you look at the last 3 testcases i added
uses the same config and ics, run w the same time, but seen from different timezones, which produce different event display times

[sdetweil]

ok, these last are cause by node-ical, mis parsing the date/time.

27th is DST switch, 3am local time, text correct here
the repeatingteventweekaftertoday

[2024-10-20 14:50:55.130] [DEBUG] parsed data={
  "[email protected]": {
    "type": "VEVENT",
    "params": [],
    "start": "2024-10-28T00:00:00.000Z",    <------   sb 1027T220000
    "datetype": "date-time",
    "end": "2024-10-28T00:00:00.000Z",    <-----     sb 1027T230000

the ics says

DTSTART;TZID=Europe/Berlin:20241028T000000
DTEND;TZID=Europe/Berlin:20241028T010000

to utc time should be -2 hours (node-ical is in utc only)

the oct 21 event

    "params": [],
    "start": "2024-10-21T00:00:00.000Z", <----- oops 
    "datetype": "date-time",
    "end": "2024-10-20T23:00:00.000Z",  <---- good

ics

DTSTART;TZID=Europe/Berlin:20241021T000000  
DTEND;TZID=Europe/Berlin:20241021T010000    

[evroom]

Just to avoid confusion, here summertime ends on Sunday October 27th, at 03:00 CEST. Going to 02:00 CET.
I made a mental note when testing, but then of course forgot about it when adding a (repeating) event on Oct 28th :-)

[sdetweil]

yes.. sunday morning dst change.. stil ical parses incorrectly

11pm event is still at 11pm (on the wall clock)

[sdetweil]

moment-timezone messed it up
in node-ical

[2024-10-20 15:43:39.018] [LOG]   tz found= Europe/Berlin  tz= Europe/Berlin  value= 20241028T000000 
[2024-10-20 15:43:39.022] [LOG]   date after moment= 2024-10-27T23:00:00.000Z 
[2024-10-20 15:43:39.022] [LOG]   date after add tz= 2024-10-27T23:00:00.000Z { tz: 'Europe/Berlin' } 
[2024-10-20 15:43:39.023] [LOG]   tz found= Europe/Berlin  tz= Europe/Berlin  value= 20241028T010000 
[2024-10-20 15:43:39.023] [LOG]   date after moment= 2024-10-28T00:00:00.000Z 
[2024-10-20 15:43:39.023] [LOG]   date after add tz= 2024-10-28T00:00:00.000Z { tz: 'Europe/Berlin' } 

[2024-10-20 15:43:39.025] [LOG]   tz found= Europe/Berlin  tz= Europe/Berlin  value= 20241021T000000 
[2024-10-20 15:43:39.025] [LOG]   date after moment= 2024-10-20T22:00:00.000Z 
[2024-10-20 15:43:39.025] [LOG]   date after add tz= 2024-10-20T22:00:00.000Z { tz: 'Europe/Berlin' } 
[2024-10-20 15:43:39.026] [LOG]   tz found= Europe/Berlin  tz= Europe/Berlin  value= 20241021T010000 
[2024-10-20 15:43:39.026] [LOG]   date after moment= 2024-10-20T23:00:00.000Z 
[2024-10-20 15:43:39.026] [LOG]   date after add tz= 2024-10-20T23:00:00.000Z { tz: 'Europe/Berlin' }

[sdetweil]

node ical bug, fixed temp

git pull
rm -rf node_modules
npm run install-mm

[evroom]

Sam,

I have tested the repeating events, using different times and dates (before, during and after the transition to wintertime, and ran some sanity checks.
To me all seems well now.

Never ran any tests, however, with dates and times in different timezones (like starting in Germany/Berlin and ending in America/New_York).

Thanks for getting to the bottom of this and admire your perseverance!!

E.J.

[sdetweil]

awesome. thanks for the great teastcases
i have 3 or 4 more to add to the testruns to make sure it stays fixed

[sdetweil]

for tz date tests, just change your system timezone

[evroom]

I more meant, that, when, for example, flying from Berlin to New York and will enter the Berlin date & time for departure and the New York date & time for arrival.
You will have 2 timezones in 1 event, right ?
Never added an event like that before and therefore never made changes to such an event.
Around a summertime / wintertime adjustment this could be fun, but perhaps it isn't that trivial.

Perhaps I will just give it a try ...

[sdetweil]

i think an event can only be in one tz

at least google cal only lets me pick one tz

you can change it after takeoff , so the landing time is correct for destination

[evroom]

I have one more question.

My MagicMirror that I use for daily life is running on a Raspberry Pi 3 Model B Plus with Raspian 'buster' (armv7l) and still is on MM version 2.24.0.
Node on version 16.20.2 and rpm on version 8.19.4.
Because of the fact that I use omxplayer for my camera feed.
I never could find a suitable player for rtsp streams, so I will stay on 'buster' till there is one (probably 'forever').

The changes you made for the calendar module, are they back-portable ?
I suppose either it works or not, right ?

Or should I go to MM version 2.29 and apply your changes like I did on my test-bench ?
Hoping that omxplayer will still work and will show my feed.

[sdetweil]

calendar should work, make a folder there called save
move the calendar files there

copy the new version files in and give it a try

new MagicMirror version will not work on buster. new nodejs and electron need a library not available

i do not think omxplayer works on bullseye or bookworm

[sdetweil]

actually, we dropped a lib calendarfetcher.js uses

[evroom]

Okay, thx, will do.

Omxplayer cannot even be installed on everything newer than buster.
So similar as MagicMirror.
But if calendar will work okay, then I am fine with that.
Only use a handful of modules on that MagicMirror.

[sdetweil]

I tried to make a recurring event w diff timezones, google cal said recurring must be same timezome

so your travel one could be starting at 10am Berlin TZ, ending at 6pm LA timezone. same of diff days..

I will test that, works good