Add form load time to xforms #34819

Charl1996 · 2024-06-25T15:01:04Z

Review by commit. The vast majority of the files are test updates, so don't panic and simply review by commit.

Product Description

No visible changes.

Technical Summary

A new form meta field is added, formLoadTime, which tracks how long a form takes to load. The mobile work has been done by Ahmad, while this is the HQ part.

To generate the ES migration I ran the following command:

./manage.py make_elastic_migration --update-index forms:form

Feature Flag

No specific FF

Safety Assurance

Safety story

Tested locally. Staging testing to be done.

Automated test coverage

No automated testing yet

QA Plan

No QA planned. QA will ensue as part of this PR.

Migrations

The migrations in this code can be safely applied first independently of the code

Rollback instructions

This PR can be reverted after deploy with no further considerations

This PR contains an elastic search mapping update migration which, according to my local env, is not reversible as such.

Labels & Review

Risk label is set correctly
The set of people pinged as reviewers is appropriate for the level of risk of the change

mkangia · 2024-06-25T15:52:37Z

corehq/apps/es/migrations/0013_add_formLoadTime_to_form_meta.py

+            name=index_runtime_name('forms-20230524'),
+            type_='xform',
+            properties={
+                'form': {'dynamic': False, 'properties': {'#type': {'type': 'keyword'}, '@name': {'type': 'keyword'}, 'case': {'dynamic': False, 'properties': {'@case_id': {'type': 'keyword'}, '@date_modified': {'format': "epoch_millis||yyyy-MM-dd||yyyy-MM-dd'T'HH:mm:ssZZ||yyyy-MM-dd'T'HH:mm:ss.SSSSSS||yyyy-MM-dd'T'HH:mm:ss.SSSSSS'Z'||yyyy-MM-dd'T'HH:mm:ss'Z'||yyyy-MM-dd'T'HH:mm:ssZ||yyyy-MM-dd'T'HH:mm:ssZZ'Z'||yyyy-MM-dd'T'HH:mm:ss.SSSZZ||yyyy-MM-dd'T'HH:mm:ss||yyyy-MM-dd' 'HH:mm:ss||yyyy-MM-dd' 'HH:mm:ss.SSSSSS||mm/dd/yy' 'HH:mm:ss", 'type': 'date'}, '@user_id': {'type': 'keyword'}, '@xmlns': {'type': 'keyword'}, 'case_id': {'type': 'keyword'}, 'date_modified': {'format': "epoch_millis||yyyy-MM-dd||yyyy-MM-dd'T'HH:mm:ssZZ||yyyy-MM-dd'T'HH:mm:ss.SSSSSS||yyyy-MM-dd'T'HH:mm:ss.SSSSSS'Z'||yyyy-MM-dd'T'HH:mm:ss'Z'||yyyy-MM-dd'T'HH:mm:ssZ||yyyy-MM-dd'T'HH:mm:ssZZ'Z'||yyyy-MM-dd'T'HH:mm:ss.SSSZZ||yyyy-MM-dd'T'HH:mm:ss||yyyy-MM-dd' 'HH:mm:ss||yyyy-MM-dd' 'HH:mm:ss.SSSSSS||mm/dd/yy' 'HH:mm:ss", 'type': 'date'}, 'user_id': {'type': 'keyword'}, 'xmlns': {'type': 'keyword'}}}, 'meta': {'dynamic': False, 'properties': {'appVersion': {'type': 'keyword'}, 'app_build_version': {'type': 'keyword'}, 'commcare_version': {'type': 'keyword'}, 'deviceID': {'type': 'keyword'}, 'formLoadTime': {'type': 'keyword'}, 'geo_point': {'type': 'geo_point'}, 'instanceID': {'type': 'keyword'}, 'timeEnd': {'format': "epoch_millis||yyyy-MM-dd||yyyy-MM-dd'T'HH:mm:ssZZ||yyyy-MM-dd'T'HH:mm:ss.SSSSSS||yyyy-MM-dd'T'HH:mm:ss.SSSSSS'Z'||yyyy-MM-dd'T'HH:mm:ss'Z'||yyyy-MM-dd'T'HH:mm:ssZ||yyyy-MM-dd'T'HH:mm:ssZZ'Z'||yyyy-MM-dd'T'HH:mm:ss.SSSZZ||yyyy-MM-dd'T'HH:mm:ss||yyyy-MM-dd' 'HH:mm:ss||yyyy-MM-dd' 'HH:mm:ss.SSSSSS||mm/dd/yy' 'HH:mm:ss", 'type': 'date'}, 'timeStart': {'format': "epoch_millis||yyyy-MM-dd||yyyy-MM-dd'T'HH:mm:ssZZ||yyyy-MM-dd'T'HH:mm:ss.SSSSSS||yyyy-MM-dd'T'HH:mm:ss.SSSSSS'Z'||yyyy-MM-dd'T'HH:mm:ss'Z'||yyyy-MM-dd'T'HH:mm:ssZ||yyyy-MM-dd'T'HH:mm:ssZZ'Z'||yyyy-MM-dd'T'HH:mm:ss.SSSZZ||yyyy-MM-dd'T'HH:mm:ss||yyyy-MM-dd' 'HH:mm:ss||yyyy-MM-dd' 'HH:mm:ss.SSSSSS||mm/dd/yy' 'HH:mm:ss", 'type': 'date'}, 'userID': {'null_value': '__NULL__', 'type': 'keyword'}, 'username': {'type': 'keyword'}}}}},


oh wow, thats something.

mkangia · 2024-06-25T15:53:54Z

corehq/apps/app_manager/xform.py

@@ -1504,6 +1508,7 @@ def _add_meta_2(self, form):
            '{orx}instanceID',
            '{cc}appVersion',
            '{orx}drift',
+            '{orx}formLoadTime',


this is going to break lots of tests! Only way is fixing them.

mkangia · 2024-06-25T15:54:58Z

I see this is still in draft, happy to take a look once ready for review.
I would also recommend we run this by someone who understands the ES rollout well.

AmitPhulera · 2024-06-26T05:18:50Z

corehq/apps/es/migrations/0013_add_formLoadTime_to_form_meta.py

+            properties={
+                'form': {'dynamic': False, 'properties': {'#type': {'type': 'keyword'}, '@name': {'type': 'keyword'}, 'case': {'dynamic': False, 'properties': {'@case_id': {'type': 'keyword'}, '@date_modified': {'format': "epoch_millis||yyyy-MM-dd||yyyy-MM-dd'T'HH:mm:ssZZ||yyyy-MM-dd'T'HH:mm:ss.SSSSSS||yyyy-MM-dd'T'HH:mm:ss.SSSSSS'Z'||yyyy-MM-dd'T'HH:mm:ss'Z'||yyyy-MM-dd'T'HH:mm:ssZ||yyyy-MM-dd'T'HH:mm:ssZZ'Z'||yyyy-MM-dd'T'HH:mm:ss.SSSZZ||yyyy-MM-dd'T'HH:mm:ss||yyyy-MM-dd' 'HH:mm:ss||yyyy-MM-dd' 'HH:mm:ss.SSSSSS||mm/dd/yy' 'HH:mm:ss", 'type': 'date'}, '@user_id': {'type': 'keyword'}, '@xmlns': {'type': 'keyword'}, 'case_id': {'type': 'keyword'}, 'date_modified': {'format': "epoch_millis||yyyy-MM-dd||yyyy-MM-dd'T'HH:mm:ssZZ||yyyy-MM-dd'T'HH:mm:ss.SSSSSS||yyyy-MM-dd'T'HH:mm:ss.SSSSSS'Z'||yyyy-MM-dd'T'HH:mm:ss'Z'||yyyy-MM-dd'T'HH:mm:ssZ||yyyy-MM-dd'T'HH:mm:ssZZ'Z'||yyyy-MM-dd'T'HH:mm:ss.SSSZZ||yyyy-MM-dd'T'HH:mm:ss||yyyy-MM-dd' 'HH:mm:ss||yyyy-MM-dd' 'HH:mm:ss.SSSSSS||mm/dd/yy' 'HH:mm:ss", 'type': 'date'}, 'user_id': {'type': 'keyword'}, 'xmlns': {'type': 'keyword'}}}, 'meta': {'dynamic': False, 'properties': {'appVersion': {'type': 'keyword'}, 'app_build_version': {'type': 'keyword'}, 'commcare_version': {'type': 'keyword'}, 'deviceID': {'type': 'keyword'}, 'formLoadTime': {'type': 'keyword'}, 'geo_point': {'type': 'geo_point'}, 'instanceID': {'type': 'keyword'}, 'timeEnd': {'format': "epoch_millis||yyyy-MM-dd||yyyy-MM-dd'T'HH:mm:ssZZ||yyyy-MM-dd'T'HH:mm:ss.SSSSSS||yyyy-MM-dd'T'HH:mm:ss.SSSSSS'Z'||yyyy-MM-dd'T'HH:mm:ss'Z'||yyyy-MM-dd'T'HH:mm:ssZ||yyyy-MM-dd'T'HH:mm:ssZZ'Z'||yyyy-MM-dd'T'HH:mm:ss.SSSZZ||yyyy-MM-dd'T'HH:mm:ss||yyyy-MM-dd' 'HH:mm:ss||yyyy-MM-dd' 'HH:mm:ss.SSSSSS||mm/dd/yy' 'HH:mm:ss", 'type': 'date'}, 'timeStart': {'format': "epoch_millis||yyyy-MM-dd||yyyy-MM-dd'T'HH:mm:ssZZ||yyyy-MM-dd'T'HH:mm:ss.SSSSSS||yyyy-MM-dd'T'HH:mm:ss.SSSSSS'Z'||yyyy-MM-dd'T'HH:mm:ss'Z'||yyyy-MM-dd'T'HH:mm:ssZ||yyyy-MM-dd'T'HH:mm:ssZZ'Z'||yyyy-MM-dd'T'HH:mm:ss.SSSZZ||yyyy-MM-dd'T'HH:mm:ss||yyyy-MM-dd' 'HH:mm:ss||yyyy-MM-dd' 'HH:mm:ss.SSSSSS||mm/dd/yy' 'HH:mm:ss", 'type': 'date'}, 'userID': {'null_value': '__NULL__', 'type': 'keyword'}, 'username': {'type': 'keyword'}}}}},
+            },
+            es_versions=[],


Can we specify version too? We have an option to specify es_version in elastic_make_migration command, it would be helpful in keeping the changes to the current version itself. In future when we move to a newer es version then this change would automatically would be incorported in the newer mappings.

Suggested change

es_versions=[],

es_versions=[5],

Charl1996 · 2024-06-26T06:48:07Z

(patiently going through fixing up the tests)

AmitPhulera · 2024-06-26T07:57:04Z

corehq/apps/es/migrations/0013_add_formLoadTime_to_form_meta.py

+    ]
+
+    operations = [
+        corehq.apps.es.migration_operations.UpdateIndexMapping(


Ahh! Another thing, since we have created the new indices that we would using for ES 6 migration, you would need to add another UpdateIndexMapping block manually in operations array.

It would look something like

corehq.apps.es.migration_operations.UpdateIndexMapping( name=index_runtime_name('forms-2024-05-09'), type_='xform', properties={ .... }, es_versions=[], )

forms-2024-05-09 is the name of new index. You can find the info here

Charl1996 · 2024-06-26T10:30:22Z

Closing in favour of #34824 (which have a better looking commit history)

mkangia · 2024-06-26T10:58:17Z

Good call @Charl1996

Charl1996 added 6 commits June 25, 2024 15:28

Prefactor: fix linting issues

4dc9ea4

Add formLoadTime to app meta

2c745c3

Add formLoadTime to XFormInstance model and XFormPhoneMetadata

e6cf73b

Add formLoadTime to form meta

077ea13

./manage.py make_elastic_migration --update-index forms:form

0bb203b

Add index runtime name

28f5def

Charl1996 requested review from mkangia and AmitPhulera June 25, 2024 15:01

dimagimon added reindex/migration Reindex or migration will be required during or before deploy Risk: High Change affects files that have been flagged as high risk. labels Jun 25, 2024

Charl1996 mentioned this pull request Jun 25, 2024

Add formLoadTime column to report #34817

Open

3 tasks

mkangia reviewed Jun 25, 2024

View reviewed changes

AmitPhulera reviewed Jun 26, 2024

View reviewed changes

Charl1996 added 3 commits June 26, 2024 08:14

Add ES version to migration

5e9f8da

Add <orx:formLoadTime/> to test suit files

eb23f2c

Fix tests

0f16a8f

Charl1996 added 3 commits June 26, 2024 08:59

Make more test xml updates

5526299

Add form_load_time to expected_form_sql.json

280ef0a

Fix couchforms tests

0c69a69

AmitPhulera reviewed Jun 26, 2024

View reviewed changes

Charl1996 added 3 commits June 26, 2024 10:36

Fix reports tests

e9817ca

Fix more couchforms

32df2c3

Add secondary index update operation in migration file

3ebb10f

Charl1996 mentioned this pull request Jun 26, 2024

Add form load time to xforms and es forms mapping #34824

Open

4 tasks

Charl1996 closed this Jun 26, 2024

mkangia deleted the add-form-load-time-to-es branch June 26, 2024 10:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add form load time to xforms #34819

Add form load time to xforms #34819

Charl1996 commented Jun 25, 2024 •

edited

Loading

mkangia Jun 25, 2024

mkangia Jun 25, 2024

mkangia commented Jun 25, 2024

AmitPhulera Jun 26, 2024

Charl1996 commented Jun 26, 2024

AmitPhulera Jun 26, 2024

Charl1996 commented Jun 26, 2024

mkangia commented Jun 26, 2024

Add form load time to xforms #34819

Add form load time to xforms #34819

Conversation

Charl1996 commented Jun 25, 2024 • edited Loading

Product Description

Technical Summary

Feature Flag

Safety Assurance

Safety story

Automated test coverage

QA Plan

Migrations

Rollback instructions

Labels & Review

mkangia Jun 25, 2024

Choose a reason for hiding this comment

mkangia Jun 25, 2024

Choose a reason for hiding this comment

mkangia commented Jun 25, 2024

AmitPhulera Jun 26, 2024

Choose a reason for hiding this comment

Charl1996 commented Jun 26, 2024

AmitPhulera Jun 26, 2024

Choose a reason for hiding this comment

Charl1996 commented Jun 26, 2024

mkangia commented Jun 26, 2024

Charl1996 commented Jun 25, 2024 •

edited

Loading