Make the Dashboard tab configurable, and implement new components #1142

JGreenlee · 2024-04-03T06:16:07Z

Up to now, the dashboard has only been shown for MULTILABEL configurations; on ENKETO configurations it was completely hidden. We are making the dashboard more configurable - e-mission/e-mission-docs#1055. The presence of a new field `metrics`.`phone_dashboard_ui` being defined will cause the dashboard to be shown. If `phone_dashboard_ui` is not defined, it will fall back to the current behavior (which is to only show dashboard for MULTILABEL configurations).

e-mission/e-mission-docs#1055 (comment) *** sections A list of sections to show in the dashboard UI. They will appear in the specified order from top to bottom. Options are footprint, active_travel, summary, engagement, surveys. Whichever sections are omitted will not be shown in the UI. Default: ["footprint", "active_travel", "summary"] ***

If there is no active_travel_options.modes_list, set the default active modes

jiji14 · 2024-04-05T20:55:43Z

[ Tasks ]

footprint_options . unlabeled_uncertainty
active_travel_options . modes_list
summary_options . metrics_list
Leaflet Map bug fix
Leaderboard - UI
Leaderboard - Data fetching

…enlee/e-mission-phone into configurable-app-dashboard

JGreenlee · 2024-04-17T05:33:11Z

I resolved the merge conflicts on #1138.
@jiji14 When you begin working on this again, you can pull from JGreenlee:refactoring_timelinecontext and the majority of the merge conflicts should go away

… configurable-app-dashboard

jiji14 · 2024-04-25T01:05:07Z

Surveys UI screenshots

Leaflet map encounters an error when prerendered, so we need to render the TimelineScrollList component when the active tab is 'label' 'shouldUpdateTimeline' state is used to determine whether to render the TimelineScrollList or not

purpose : prevent too much API call

jiji14 · 2024-05-05T00:35:38Z

@JGreenlee I have completed the frontend code for the survey dashboard. Currently, I am still working on the server code, and I will provide an explanation in the server pull request.

I have implemented a new API to fetch survey data every 24 hours because the survey data does not render based on the timeline. It should display the accumulated results. Therefore, I designed it to call the survey API when a user clicks on the dashboard tab and the last updated data was more than 24 hours ago.

Additionally, after retrieving survey data, I preprocess it for charts. Regarding the leaderboard, I have configured it to display only the top 5 members, myself, and the bottom 3 members.

Please review the code and comment if anything doesn't make sense to you. I will bring my personal laptop to Hawaii so I can communicate via GitHub. Thank you!

…enlee/e-mission-phone into configurable-app-dashboard

When I first designed the survey dashboard, it showed the accumulated survey data regardless of the selected date range. However, the updated logic is that the survey metric depends on the date range selected, while the leaderboard metric remains accumulated.

Survey Comparison Card: Data depends on date range. Survey Leaderboard Card: Data is accumulated. To prevent confusion for users, we separate the two cards and inform them that the leaderboard data is accumulated.

jiji14 · 2024-05-16T21:52:57Z

[ Upate ]
Survey Comparison Card: Data depends on date range.
Survey Leaderboard Card: Data is accumulated.

To prevent confusion for users, we separate the two cards and inform them that the leaderboard data is accumulated.

Need to fix fetching data after the server side is done

Instead of querying by timestamp we can now query by "YYYY-MM-DD" strings. The yyyy_mm_dd endpoint also has a different format for metric_list: instead of just an array of metrics, it is a mapping of "metric names" to "grouping fields" (e-mission/e-mission-server#966 (comment)) We will specify this in the config as "metrics_list". The yyyy_mm_dd endpoint will also ask for app_config, so we will pass that through to `fetchMetricsFromServer` and include it in the `query` (note that `survey_info` is the only field that needs to be included) We will be supporting another metric called "response_count"; added a card to the 'summary' section (if it is configured to show) Updated types in appConfigTypes.ts

Making this more generic because we will now support fields other than the labeled mode (such as labeled purpose, BLE sensed mode, and survey that was prompted)

Instead of specifically listing a MetricsCard for each metric, we can read metricsList and map each key to a MetricsCard. But the structure for response_count is an object of {responded: number, not_responded: number}, while the other metrics are just numbers. So we need to adjust change the way days get summed up to account for both.

The 'showsIf' conditions in dfc-fermata currently reference 'confirmedMode'. We want to rename this to 'primary_ble_sensed_mode'. Also see JGreenlee/e-mission-common@0396339

This makes it easier to test aggregate metrics locally

JGreenlee · 2024-05-24T15:27:02Z

Not bad actually. Looks like it's only metricsHelper.test.ts

`new Date(isoString)` assumes that `isoString` is in UTC. We wanted to parse it in the local timezone, which is what luxon does here.

There is no discrepancy between DayOfClientMetricData and DayOfServerMetricData anymore, so this test file gets stripped down a lot

codecov · 2024-05-24T15:50:48Z

Codecov Report

Attention: Patch coverage is 16.23377% with 258 lines in your changes are missing coverage. Please review.

Project coverage is 30.59%. Comparing base (39d463c) to head (e55e5ae).
Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1142      +/-   ##
==========================================
+ Coverage   28.92%   30.59%   +1.66%     
==========================================
  Files         115      118       +3     
  Lines        5064     5216     +152     
  Branches     1094     1154      +60     
==========================================
+ Hits         1465     1596     +131     
- Misses       3597     3618      +21     
  Partials        2        2

Flag	Coverage Δ
unit	`30.59% <16.23%> (+1.66%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
www/js/appTheme.ts	`100.00% <ø> (+100.00%)`	⬆️
www/js/components/Carousel.tsx	`100.00% <100.00%> (+100.00%)`	⬆️
www/js/config/useImperialConfig.ts	`87.09% <100.00%> (+38.82%)`	⬆️
www/js/diary/list/DateSelect.tsx	`50.00% <100.00%> (+50.00%)`	⬆️
www/js/config/dynamicConfig.ts	`84.72% <80.00%> (+0.32%)`	⬆️
www/js/survey/multilabel/confirmHelper.ts	`96.22% <50.00%> (-0.89%)`	⬇️
www/js/Main.tsx	`0.00% <0.00%> (ø)`
www/js/diary/list/FilterSelect.tsx	`0.00% <0.00%> (ø)`
www/js/services/commHelper.ts	`24.79% <0.00%> (-0.21%)`	⬇️
www/js/control/ProfileSettings.tsx	`0.00% <0.00%> (ø)`
... and 15 more

... and 3 files with indirect coverage changes

JGreenlee · 2024-05-24T16:02:16Z

We are lacking in code coverage, but tests passed

jiji14 · 2024-05-25T19:15:31Z

@JGreenlee
Thanks for your wonderful work! I fixed some minor css issues.

I added 'no trip data' message for surveyComparison chart.
[ before ]

[ after ]

Also set the maxBarThickness to make it prettier!

@shankari
Other than those CSS issues, everything went smoothly, so I think we are ready to go for staging! I will do more testing on the staging app.

shankari · 2024-05-29T06:08:25Z

I am halfway through reviewing this change (15/31 files) but they were the easy 15. Will finish review of the changes to the metrics folder tomorrow.

shankari · 2024-05-29T06:09:03Z

@jiji14 you need to re-run prettier; the test is failing

jiji14 · 2024-05-30T03:54:25Z

@JGreenlee @shankari
I added more unit tests in metricsHelper and useImperialConfig, as well as basic component tests for Carousel and DateSelect. I will continue to add more tests to achieve higher code coverage. Codecov seems to encounter a public key error again.. (The error log seems different from the previous one.) I will research what is causing it!

shankari · 2024-05-31T05:04:05Z

@jiji14 the error appears to be the same, since it was successful on re-run. However, the patch check is still failing since only 16.23% of the newly added code. The project code coverage has gone up from 29% to 31%, but the patch (aka newly added code) is still not tested enough.

shankari

I am fine with the implementation, but have a lot of comments on the tests. I am merging this now so that we can get it onto staging, but @jiji14, please address the test comments, and improve the code coverage of this patch in a follow-up PR

shankari · 2024-05-29T06:05:23Z

www/js/services/commHelper.ts

  return new Promise((rs, rj) => {
+    // when app config does not have "server", localhost is used and no user authentication is required
+    serverConnConfig ||= {
+      connectUrl: 'http://localhost:8080' as any,


nit: this is different for android and iOS. localhost only works for iOS emulators, for android emulators, the laptop it is running on is 10.0.2.2

shankari · 2024-05-31T02:49:59Z

www/js/metrics/CarbonFootprintCard.tsx

+  // Whether to show the uncertainty on the carbon footprint charts, default: true
+  const showUnlabeledMetrics =
+    appConfig?.metrics?.phone_dashboard_ui?.footprint_options?.unlabeled_uncertainty ?? true;


why is this a config option? I assume we would always have uncertainty unless every trip is labeled.
We don't want to give people a false sense of certainty.
I found the related commit jiji14@5130653 but it doesn't have any details 😄

shankari · 2024-05-31T04:06:06Z

www/js/metrics/SurveyLeaderboardCard.tsx

I am reviewing this, although I assume it will be modified when we enable the leaderboard

shankari · 2024-05-31T04:26:52Z

www/js/types/appConfigTypes.ts

+    };
+    summary_options?: {};
+    engagement_options?: {
+      leaderboard_metric: [string, string];


this doesn't appear to be used anywhere now, I assume this will also be used when we enable the leaderboard

shankari · 2024-05-31T04:50:26Z

www/js/metrics/metricsHelper.ts

+export const trimGroupingPrefix = (label: string) => {
+  for (let field of groupingFields) {
+    if (label.startsWith(field)) {
+      return label.substring(field.length + 1);
+    }
+  }


when we implement e-mission/e-mission-server#966 (comment) this can be much simplified.

shankari · 2024-05-31T05:26:32Z

www/__tests__/metricsHelper.test.ts

-    const days2 = [
-      { mode_a: 1, mode_b: 2 },
-      { mode_c: 1, mode_d: 3 },
-    ] as any as DayOfClientMetricData[];
-    it("should return unique labels for days with 'mode_*'", () => {
-      expect(getUniqueLabelsForDays(days2)).toEqual(['a', 'b', 'c', 'd']);
-    });


@jiji14 please replace this with a test that actually have duplicate labels (see my comment in #1138 (comment))

shankari · 2024-05-31T05:28:06Z

www/__tests__/metricsHelper.test.ts

-
-    const days2 = [
-      { fmt_time: '2021-01-01T00:00:00Z' },
-      { fmt_time: '2021-01-02T00:00:00Z' },
-      { fmt_time: '2021-01-04T00:00:00Z' },
-      { fmt_time: '2021-01-08T00:00:00Z' },
-      { fmt_time: '2021-01-09T00:00:00Z' },
-      { fmt_time: '2021-01-10T00:00:00Z' },
-    ] as any as DayOfServerMetricData[];
-    it("should segment days with 'fmt_time' into weeks", () => {
-      expect(segmentDaysByWeeks(days2, '2021-01-10')).toEqual([
-        // most recent week
-        [
-          { fmt_time: '2021-01-04T00:00:00Z' },
-          { fmt_time: '2021-01-08T00:00:00Z' },
-          { fmt_time: '2021-01-09T00:00:00Z' },
-          { fmt_time: '2021-01-10T00:00:00Z' },
-        ],
-        // prior week
-        [{ fmt_time: '2021-01-01T00:00:00Z' }, { fmt_time: '2021-01-02T00:00:00Z' }],
-      ]);
-    });


Again, please fix this test that was removed to get the tests to passs

shankari · 2024-05-31T05:31:32Z

www/__tests__/metricsHelper.test.ts

+      expect(result).toBe(true);
+    });
+
+    it('returns true for all sensed labels', () => {


for sensed labels, we expect isCustomLabels to return false. Right now, this is identical to the previous test.

shankari · 2024-05-31T05:54:45Z

www/__tests__/metricsHelper.test.ts

+    it('checks for count metric', () => {
+      const result = getUnitUtilsForMetric('count', imperialConfig);
+      expect(result).toEqual(['trips', expect.any(Function), expect.any(Function)]);
+      const mockTrip = { responded: 4, not_responded: 3 };
+      expect(result[1](mockTrip)).toBe(mockTrip);
+      expect(result[2](mockTrip)).toBe(mockTrip + ' trips');
+    });


This passes, but I don't think it is testing the right functionality. count is a metric to just count the number of trips. The metric that expects a structure with responded and not_responded is response_count

shankari · 2024-05-31T05:57:22Z

www/__tests__/useImperialConfig.test.ts

@@ -53,3 +61,15 @@ describe('convertSpeed', () => {
    expect(convertSpeed(6.7056, true)).toBeCloseTo(15); // Approximately 15 mph
  });
 });
+
+describe('useImperialConfig', () => {
+  it('returns ImperialConfig with imperial units', () => {


Imperial units are the old ones (popularized by the British Empire), so miles, mph etc. These are non-imperial or SI units. Shouldn't we also add a similar check where we expect to see miles, mph etc?

shankari · 2024-05-31T06:54:49Z

This is now on staging as version 1.7.9. The staging server has also been updated to incorporate

Nominatim test regression fix
Nominatim test regression fix e-mission-server#967
make localdate queries behave like canonical datetime range filters
make localdate queries behave like canonical datetime range filters e-mission-server#968
make push notifs configurable, i18n, & only for users missing labels
make push notifs configurable, i18n, & only for users missing labels e-mission-server#969
Dashboard Survey Metrics
Dashboard Survey Metrics e-mission-server#966

A couple of issues that encountered while testing before pushing to staging were

Error while loading aggregate data	Only loads trips for the past 14 days, if there are no trips in that range, unlike before, nothing is loaded. How does the user know how far back they need to go?

shankari · 2024-05-31T20:50:51Z

500 error has been resolved.

@JGreenlee I still see the following issues

Only loads trips for the last 14 days	Group bar not displayed

Note also that the active minutes table shows overlapping ranges (two weeks, then two one weeks and then per day). Not sure if that is intended, and it was a bit confusing when I first looked at it.

jiji14 · 2024-05-31T21:47:25Z

@shankari
Thank you for all the comments on testing! I will create a new PR to address the issues you raised, adding more tests and fixing any issues identified. I agree that the component testing is too basic. I've been researching how to mock useState for click events, as component tests continue to fail due to this issue. I will keep digging into it!

JGreenlee · 2024-06-03T14:16:56Z

@JGreenlee I still see the following issues

I am looking into these issues this morning

When refactoring into TimelineContext, there was a regression regarding the date range that is initially loaded. e-mission#1142 (comment) It used to load 7 days before the pipelineEnd up to today. After the change, it would load 7 days before today up to today. If the user has travel everyday this ends up being equivalent. But if there isn't recent travel, this becomes a problem. So in TimelineContext, the initial dateRange needs to depend on pipelineRange. instead of giving an initial value upfront, we'll let dateRange be null until pipelineRange is set, at which point we'll also set dateRange. This means we have to consider some extra cases where dateRange can be null.

JGreenlee and others added 4 commits April 3, 2024 02:12

show/hide unlabeled metrics based on config

5130653

set the active modes based on config (active_travel_options.modes_list)

16edaf2

If there is no active_travel_options.modes_list, set the default active modes

jiji14 and others added 2 commits April 5, 2024 15:10

set summary list based on config (summary_options.metrics_list)

b17ffb7

Merge branch 'refactoring_timelinecontext' of https://github.com/JGre…

21adb20

…enlee/e-mission-phone into configurable-app-dashboard

jiji14 added 5 commits April 18, 2024 12:08

Merge remote-tracking branch 'other/refactoring_timelinecontext' into…

84f8a51

… configurable-app-dashboard

show uncertain-footnote only when to show the uncertainty

b95c268

rendering surveys Card components dynamically

57d2298

Surveys Cards components UI done

18e99bb

Chart UI fix

2281c7e

jiji14 added 4 commits April 25, 2024 17:15

add getSurveyMetric API endpoint

1f7a692

get new survey data every 24 hours

58cfb5f

purpose : prevent too much API call

delete dummy data and process survey data from server

8150e95

jiji14 mentioned this pull request May 5, 2024

Dashboard Survey Metrics e-mission/e-mission-server#966

Merged

JGreenlee and others added 3 commits May 13, 2024 12:27

Merge branch 'refactoring_timelinecontext' of https://github.com/JGre…

ed52e0f

…enlee/e-mission-phone into configurable-app-dashboard

Separate 'Survey comparison Card' and 'Leaderboard Card'

d4b3f51

Survey Comparison Card: Data depends on date range. Survey Leaderboard Card: Data is accumulated. To prevent confusion for users, we separate the two cards and inform them that the leaderboard data is accumulated.

JGreenlee added 6 commits May 20, 2024 12:28

refactor valueForModeOnDay -> valueForFieldOnDay

5ab984c

Making this more generic because we will now support fields other than the labeled mode (such as labeled purpose, BLE sensed mode, and survey that was prompted)

include primary_ble_sensed_mode in derived properties

ab6fece

The 'showsIf' conditions in dfc-fermata currently reference 'confirmedMode'. We want to rename this to 'primary_ble_sensed_mode'. Also see JGreenlee/e-mission-common@0396339

support localhost metrics when appConfig does not have 'server'

c9628e0

This makes it easier to test aggregate metrics locally

add 'metrics.responses' to en.json

32336ea

JGreenlee added 2 commits May 24, 2024 11:37

fix date range picker timezone issue

b3e80df

`new Date(isoString)` assumes that `isoString` is in UTC. We wanted to parse it in the local timezone, which is what luxon does here.

update metricsHelper.test.ts to use the unified DayOfMetricData type

663f86c

There is no discrepancy between DayOfClientMetricData and DayOfServerMetricData anymore, so this test file gets stripped down a lot

jiji14 added 4 commits May 25, 2024 00:06

Hide chart when there is no survey data

90b631c

type error handling

a3570d6

fix checking data logic for surveyComparisonCard

084224e

Add maxBarThickness for chart.js

8cf5fb0

jiji14 added 2 commits May 28, 2024 15:15

add more unit tests

f55faf1

export functions for tests and fix syntax issue

abf2f3f

jiji14 added 5 commits May 29, 2024 03:05

done unit tests with metricsHelper

66464ca

add useImperialConfig test

a963381

add getTheme unit test

3f08b6c

add unit test if Carousel renders children correctly

1bd1f48

add DateSelect rendering unit test

e55e5ae

shankari requested changes May 31, 2024

View reviewed changes

shankari merged commit 461e5fe into e-mission:master May 31, 2024
7 of 8 checks passed

jiji14 mentioned this pull request Jun 7, 2024

Unit Test for new Dashboard feature #1161

Open

JGreenlee mentioned this pull request Jun 7, 2024

fix timeline initial date range regression #1162

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make the Dashboard tab configurable, and implement new components #1142

Make the Dashboard tab configurable, and implement new components #1142

JGreenlee commented Apr 3, 2024

jiji14 commented Apr 5, 2024 •

edited

Loading

JGreenlee commented Apr 17, 2024

jiji14 commented Apr 25, 2024

jiji14 commented May 5, 2024

jiji14 commented May 16, 2024

JGreenlee commented May 24, 2024

codecov bot commented May 24, 2024 •

edited

Loading

JGreenlee commented May 24, 2024

jiji14 commented May 25, 2024

shankari commented May 29, 2024

shankari commented May 29, 2024

jiji14 commented May 30, 2024

shankari commented May 31, 2024

shankari left a comment •

edited

Loading

shankari May 29, 2024

shankari May 31, 2024

shankari May 31, 2024

shankari May 31, 2024

shankari May 31, 2024

shankari May 31, 2024

shankari May 31, 2024

shankari May 31, 2024

shankari May 31, 2024

shankari May 31, 2024

shankari commented May 31, 2024

shankari commented May 31, 2024

jiji14 commented May 31, 2024

JGreenlee commented Jun 3, 2024

Make the Dashboard tab configurable, and implement new components #1142

Make the Dashboard tab configurable, and implement new components #1142

Conversation

JGreenlee commented Apr 3, 2024

jiji14 commented Apr 5, 2024 • edited Loading

JGreenlee commented Apr 17, 2024

jiji14 commented Apr 25, 2024

Surveys UI screenshots

jiji14 commented May 5, 2024

jiji14 commented May 16, 2024

JGreenlee commented May 24, 2024

codecov bot commented May 24, 2024 • edited Loading

Codecov Report

JGreenlee commented May 24, 2024

jiji14 commented May 25, 2024

shankari commented May 29, 2024

shankari commented May 29, 2024

jiji14 commented May 30, 2024

shankari commented May 31, 2024

shankari left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shankari commented May 31, 2024

shankari commented May 31, 2024

jiji14 commented May 31, 2024

JGreenlee commented Jun 3, 2024

jiji14 commented Apr 5, 2024 •

edited

Loading

codecov bot commented May 24, 2024 •

edited

Loading

shankari left a comment •

edited

Loading