Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add installation truncation #192

Closed
wants to merge 134 commits into from
Closed
Show file tree
Hide file tree
Changes from 128 commits
Commits
Show all changes
134 commits
Select commit Hold shift + click to select a range
977309e
add logic to write status complete file
feng-j678 May 10, 2023
012fb39
add logic for latest status complete file
feng-j678 May 10, 2023
86adb29
add truncation logic
feng-j678 May 23, 2023
4c182ea
change logic to binary search and trying utest
feng-j678 May 29, 2023
089fac6
add utest for truncated file, change status_complete*** to complete_s…
feng-j678 May 29, 2023
6cf7d99
add unit test to CoreMain for truncation and modify the load_status_f…
feng-j678 May 29, 2023
89f2d92
add unit test for testing errors for truncation
feng-j678 May 29, 2023
6cdce2b
Merge branch 'master' of https://github.com/Azure/LinuxPatchExtension…
feng-j678 May 30, 2023
83f7d17
remove json.dumps on message to avoid adding \ and remove logic for c…
feng-j678 Jun 1, 2023
ece5ad6
add logic for quote count after truncation
feng-j678 Jun 2, 2023
f213d6f
modify the quote_count logic and set_package_assessment_status to use…
feng-j678 Jun 2, 2023
7114d6d
fix file formating and add more tests for report_sequence_number_chan…
feng-j678 Jun 2, 2023
8242685
add more unit test
feng-j678 Jun 2, 2023
faa0bbc
remove unused variables in test_status
feng-j678 Jun 2, 2023
e9800ab
add truncation logic for installation
feng-j678 Jun 5, 2023
fdf7d8f
modify the set_package_install_status and classification to use map f…
feng-j678 Jun 5, 2023
1555c07
add unit tests for installation truncation
feng-j678 Jun 6, 2023
45823ac
add more unit test for testing installation truncation
feng-j678 Jun 6, 2023
3e1cd68
reset installation_tmp_map
feng-j678 Jun 6, 2023
6d6f8fa
add classification to test_installatio_keep_5
feng-j678 Jun 6, 2023
89e99da
revert set_pacakge_install_status to double forloops because values i…
feng-j678 Jun 7, 2023
beb92bb
change hashmap to orderedmap because existing utest check for order o…
feng-j678 Jun 7, 2023
9ea154e
add unit test for get latest complete status file
feng-j678 Jun 7, 2023
e0d19b6
add test for get latest complete status file
feng-j678 Jun 7, 2023
790bc13
add descriptions to truncation tests
feng-j678 Jun 7, 2023
cfec898
add unit test for ordered map
feng-j678 Jun 7, 2023
274d69b
add test for checking if complete status is dir
feng-j678 Jun 8, 2023
7fd4383
add unit test for orderdict()
feng-j678 Jun 8, 2023
eb353e6
Modify Agent status Constants and revert white spaces
feng-j678 Jun 8, 2023
0577a4d
this has assessment truncatin logic bad format
feng-j678 Jun 8, 2023
a5c82fa
revert status handler format to as master version
feng-j678 Jun 8, 2023
fcc738e
revert code format from master version
feng-j678 Jun 8, 2023
662b176
modify variable naming convention add unit for latest complete status
feng-j678 Jun 8, 2023
65ee40d
change map to use orderdict to preserve element ordering because of py27
feng-j678 Jun 8, 2023
65cf245
add unit test for ordered map
feng-j678 Jun 8, 2023
3d186b9
chnage test_value to patch_count_for_test
feng-j678 Jun 12, 2023
4b60b38
modify the tombstone according to classifications
feng-j678 Jun 13, 2023
3d4b97f
modify unit tests for new tombstone records
feng-j678 Jun 13, 2023
4782bc3
add unit test for complete status file is a dir
feng-j678 Jun 13, 2023
0178e9d
add unit test for log_truncated_removed_packages_if_any
feng-j678 Jun 13, 2023
668aed6
merge new changes from assessment_truncation branch
feng-j678 Jun 14, 2023
3a8d421
merge in changes from assessment_truncation branch
feng-j678 Jun 14, 2023
4f4456e
resolve unit tests failures
feng-j678 Jun 18, 2023
3a27c9e
resolved unit tests problems
feng-j678 Jun 18, 2023
aed5b9a
failing unit tests
feng-j678 Jun 18, 2023
5695532
debug why unit tests are failing in coremain
feng-j678 Jun 18, 2023
5683aa5
add runtime.stop() to fix failing
feng-j678 Jun 18, 2023
d5ee671
fix unit test by correcting message patch tombstone record
feng-j678 Jun 19, 2023
d41b75b
modify unit test to whether file size is > or < agent file size limit
feng-j678 Jun 19, 2023
64aafd1
to reuse __set_errors_json for truncation
feng-j678 Jun 19, 2023
25ee9b1
remove code parameter in recompose logic
feng-j678 Jun 19, 2023
8b5c729
remove checks for agent file size limit in unit test per summary patches
feng-j678 Jun 19, 2023
8ba655b
add get_index method for low pri
feng-j678 Jun 20, 2023
3b27791
add logic for high and low pri intallation
feng-j678 Jun 22, 2023
7b8246b
add method description to __get_installation_packages_index
feng-j678 Jun 22, 2023
510a6da
extract assessment tombstone into a separate method
feng-j678 Jun 22, 2023
02fd327
refactor truncation main logic
feng-j678 Jun 22, 2023
8bbb522
merge changes from master to fix ci.yml and update statushandler
feng-j678 Jun 22, 2023
67576b3
add code flow to write_status_file()
feng-j678 Jun 22, 2023
dfcc478
refactor assessment cod, remove quote_count and use fix escape byte
feng-j678 Jun 23, 2023
88b6137
Merge branch 'master' of https://github.com/Azure/LinuxPatchExtension…
feng-j678 Jun 23, 2023
57e1cf5
refactor __recompose_truncated_status_file
feng-j678 Jun 23, 2023
141e6c1
fix code to consider no assessment truncation, everything remains same
feng-j678 Jun 23, 2023
33fd85a
update truncation exception message unit test
feng-j678 Jun 25, 2023
2c64887
remove unncessary removed_packages_details method
feng-j678 Jun 25, 2023
716d1d3
refactor truncation main logic
feng-j678 Jun 25, 2023
9f600ed
add get status file w/o list method
feng-j678 Jun 25, 2023
e4f682b
rename truncation varibles
feng-j678 Jun 25, 2023
4a559bd
use while loop to checking size limit
feng-j678 Jun 26, 2023
3023038
test fail pipeline
feng-j678 Jun 26, 2023
0c0d8da
add low_index_pri logic, status unit tests passed
feng-j678 Jun 26, 2023
8f45390
add low_pri index logic, pass all tests
feng-j678 Jun 26, 2023
f4a99c0
update unit tests and code flow
feng-j678 Jun 27, 2023
0e8650b
add extra to bottom of status handler.py
feng-j678 Jun 27, 2023
a098abc
change orderdict to collections.ordereddict
feng-j678 Jul 5, 2023
e474c34
change get latest complete status file to remove older complete status
feng-j678 Jul 6, 2023
89b88ef
change forward slash to back slash in glob.glob:
feng-j678 Jul 6, 2023
6301415
remove unused imports
feng-j678 Jul 6, 2023
6342db0
merge in changes from complete_status_file_logic branch
feng-j678 Jul 6, 2023
9004e51
pr review with todos and comments
feng-j678 Jul 26, 2023
1aa276b
commit 1 -refactor to resuse substatus_message method
feng-j678 Jul 26, 2023
e65e8b4
commit 2 - rename variables
feng-j678 Jul 26, 2023
21f6394
commit 3 - refactor global varibles assessment, installation packages…
feng-j678 Jul 26, 2023
9e428d6
Merge branch 'master' of https://github.com/Azure/LinuxPatchExtension…
feng-j678 Jul 26, 2023
e798676
change size_limit to capacity
feng-j678 Jul 27, 2023
997c758
commit 2 combine assessment and installation logic into one
feng-j678 Jul 27, 2023
7b539a1
change keep latest complete status logic to keep 10
feng-j678 Jul 27, 2023
97727f1
remove the break logic from current_list - diff
feng-j678 Jul 27, 2023
f363707
chnage test_assessment_operation_success_truncation_over_capacity bec…
feng-j678 Jul 27, 2023
f90b2f8
fix __removed_older_complete_status_files bug
feng-j678 Jul 27, 2023
2c99f29
commit 1 - refactor var, method, and combin logic in __apply_truncati…
feng-j678 Jul 28, 2023
165ef8e
commit 2 add fake version to tombstone, modify __removed_older_comple…
feng-j678 Jul 28, 2023
3d7ca29
commit 3 update pesudo code
feng-j678 Jul 28, 2023
e6ae224
commit 4 remove assertion on complete status deletion becuase github …
feng-j678 Jul 28, 2023
6a8b0f1
refactor truncation logic to write_complete_status_file()
feng-j678 Aug 4, 2023
7cf753f
commit 2 refactor substatus_msg
feng-j678 Aug 4, 2023
75afd05
commit 3 add write status comment
feng-j678 Aug 5, 2023
e8305a5
commi t 4 refactor assessment installation variables names
feng-j678 Aug 7, 2023
a89510d
Merge branch 'master' of https://github.com/Azure/LinuxPatchExtension…
feng-j678 Aug 7, 2023
df93219
commit 1 move the max_list_capacity - keepfive logic to __apply_trunc…
feng-j678 Aug 7, 2023
0623c2f
commit 2 add utf-8 encode to unit test
feng-j678 Aug 7, 2023
6e55396
commit 3 change variables in __get_new_size_in_bytes_after_truncation
feng-j678 Aug 7, 2023
f8d5e46
commit 4 simplify recompose assessment and installation substatus msg…
feng-j678 Aug 7, 2023
cd87ebc
add extra space to eof
feng-j678 Aug 7, 2023
30c05d3
commit 1 change the while loop condition to status_file_byte_size > …
feng-j678 Aug 9, 2023
9245c3b
commit 2 update pesuedo code
feng-j678 Aug 9, 2023
22d6d65
commit 3 refactor methods decriptions
feng-j678 Aug 9, 2023
ba077d2
test self.assertEqual(len(message_patches), 6) is 6 not 7
feng-j678 Aug 9, 2023
5fc2dd9
change self.assertTrue(len(message_patches) >= 6)
feng-j678 Aug 9, 2023
28d81d7
add performance for truncation
feng-j678 Aug 9, 2023
a328491
update test_installation_truncation_with_only_install_packages_over_s…
feng-j678 Aug 9, 2023
1b52a78
correct end.time - start.time in unit test
feng-j678 Aug 9, 2023
71fdd6e
remove comments and refactor __update_substatus_msg_packages_and_errors
feng-j678 Aug 10, 2023
ac0db9b
commit 1 refactor variables, methods
feng-j678 Aug 14, 2023
200e37d
commit 2 add named parameters
feng-j678 Aug 14, 2023
40eac52
refactor code
feng-j678 Aug 18, 2023
cd30132
refactor __recompose_truncated_status_file
feng-j678 Aug 18, 2023
9d083f8
merge in changes from master
feng-j678 Aug 21, 2023
f34c848
reset code changes because branch is ahead of master, refactor variab…
feng-j678 Aug 29, 2023
63351e2
add unit test for validating the log message
feng-j678 Aug 29, 2023
d775cea
change write log to tmp and read from it
feng-j678 Aug 29, 2023
c2fdc5b
move write to temp log file into a method for unit test
feng-j678 Aug 29, 2023
442593c
refact substatus_msg_at_index to substatus_message
feng-j678 Aug 29, 2023
aa7fb97
add FORCE_WRITE_TRUNCATION constant, add check for 1 min timestamp co…
feng-j678 Sep 11, 2023
7120636
add unit test for 1 min delay
feng-j678 Sep 11, 2023
764b59b
add checker for delaying truncation logic and unit test
feng-j678 Sep 28, 2023
cf4b555
Merge branch 'master' of https://github.com/Azure/LinuxPatchExtension…
feng-j678 Sep 28, 2023
51a3d53
add new comments and reset truncation boolean in load_status_file_com…
feng-j678 Sep 29, 2023
6a92ec0
refactor method names
feng-j678 Oct 3, 2023
831cb70
initialize truncation_timestamp inside __check_truncation_time_passed…
feng-j678 Oct 19, 2023
2521e8a
move self.__is_file_truncated logic to __check_status_file_size_perfo…
feng-j678 Oct 19, 2023
d322898
extract tombstone logic into a variable
feng-j678 Oct 20, 2023
2d411e1
rename var and method, and set Constants.StatusTruncationConfig.NO_TR…
feng-j678 Oct 20, 2023
424953d
change NO_TRUNCATION_IN_X_SEC = -1 to ensure truncation happens
feng-j678 Oct 20, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions src/core/src/CoreMain.py
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,7 @@ def __init__(self, argv):
patch_installer.mark_installation_completed()
overall_patch_installation_operation_successful = True
self.update_patch_substatus_if_pending(patch_operation_requested, overall_patch_installation_operation_successful, patch_assessment_successful, configure_patching_successful, status_handler, composite_logger)
status_handler.log_truncated_packages()

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

q: Wouldn't truncated packages be logged at the end of each status file write?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

koshy wanted them at the very end of operation to reduce the space on log file. I am thinking on removing this method because we are retaining up to 10 complete status.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@feng-j678 pending

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My opinion, this should be added for each status file write, post truncation, because that gives the details of what happened in each truncation (and thus status file writes). Will help in debugging issues in future. @kjohn-msft for more thoughts

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: to be raised and discussed between @kjohn-msft, @feng-j678 and me

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pending for koshy's input on this

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment is still open and needs feedback. @feng-j678 please make sure all existing comments are addressed

except Exception as error:
# Privileged operation handling for non-production use
Expand Down
11 changes: 11 additions & 0 deletions src/core/src/bootstrap/Constants.py
Original file line number Diff line number Diff line change
Expand Up @@ -176,6 +176,15 @@ class AutoAssessmentStates(EnumBackport):
STATUS_SUCCESS = "Success"
STATUS_WARNING = "Warning"

# Status file size
class StatusTruncationConfig(EnumBackport):
INTERNAL_FILE_SIZE_LIMIT_IN_BYTES = 126 * 1024
AGENT_FACING_STATUS_FILE_SIZE_LIMIT_IN_BYTES = 128 * 1024
MIN_ASSESSMENT_PACKAGE_TO_RETAIN = 5
TRUNCATION_WARNING_MESSAGE = "Package lists were truncated to limit reporting data volume. In-VM logs contain complete lists."
TURN_ON_TRUNCATION = True
SKIP_TRUNCATION_LOGIC_IN_X_SEC = 60

# Wrapper-core handshake files
EXT_STATE_FILE = 'ExtState.json'
CORE_STATE_FILE = 'CoreState.json'
Expand Down Expand Up @@ -268,13 +277,15 @@ class PatchAssessmentSummaryStartedBy(EnumBackport):
class PatchOperationTopLevelErrorCode(EnumBackport):
SUCCESS = 0
ERROR = 1
WARNING = 2

class PatchOperationErrorCodes(EnumBackport):
DEFAULT_ERROR = "ERROR" # default error code
OPERATION_FAILED = "OPERATION_FAILED"
PACKAGE_MANAGER_FAILURE = "PACKAGE_MANAGER_FAILURE"
NEWER_OPERATION_SUPERSEDED = "NEWER_OPERATION_SUPERSEDED"
UA_ESM_REQUIRED = "UA_ESM_REQUIRED"
TRUNCATION = "PACKAGE_LIST_TRUNCATED"

ERROR_ADDED_TO_STATUS = "Error_added_to_status"

Expand Down
4 changes: 1 addition & 3 deletions src/core/src/core_logic/ConfigurePatchingProcessor.py
Original file line number Diff line number Diff line change
Expand Up @@ -140,9 +140,7 @@ def __report_consolidated_configure_patch_status(self, status=Constants.STATUS_T
self.status_handler.add_error_to_status(error_msg, Constants.PatchOperationErrorCodes.DEFAULT_ERROR, current_operation_override_for_error=Constants.CONFIGURE_PATCHING_AUTO_ASSESSMENT)

# write consolidated status
self.status_handler.set_configure_patching_substatus_json(status=status,
automatic_os_patch_state=self.current_auto_os_patch_state,
auto_assessment_state=self.current_auto_assessment_state)
self.status_handler.set_configure_patching_substatus_json(status=status, automatic_os_patch_state=self.current_auto_os_patch_state, auto_assessment_state=self.current_auto_assessment_state)

def __raise_if_telemetry_unsupported(self):
if self.lifecycle_manager.get_vm_cloud_type() == Constants.VMCloudType.ARC and self.execution_config.operation not in [Constants.ASSESSMENT, Constants.INSTALLATION]:
Expand Down
Loading
Loading