Code

add installation truncation#192

Code

Closed

feng-j678 wants to merge 134 commits intomasterAzure/LinuxPatchExtension:masterfrom user/johnfeng/add_installation_truncationAzure/LinuxPatchExtension:user/johnfeng/add_installation_truncationCopy head branch name to clipboard

+1,762-78

Commits

Commits on May 10, 2023

• add logic to write status complete file

  feng-j678

  committed
  977309e

• add logic for latest status complete file

  feng-j678

  committed
  012fb39

Commits on May 23, 2023

• add truncation logic

  feng-j678

  committed
  86adb29

Commits on May 29, 2023

• change logic to binary search and trying utest

  feng-j678

  committed
  4c182ea

• add utest for truncated file, change status_complete*** to complete_status_file_path

  feng-j678

  committed
  089fac6

• add unit test to CoreMain for truncation and modify the load_status_file_component to load complete status

  feng-j678

  committed
  6cf7d99

• add unit test for testing errors for truncation

  feng-j678

  committed
  89f2d92

Commits on May 30, 2023

• Merge branch 'master' of https://github.com/Azure/LinuxPatchExtension into user/johnfeng/add_assessment_truncation

  feng-j678

  committed
  6cdce2b

Commits on Jun 1, 2023

• remove json.dumps on message to avoid adding \ and remove logic for counting quotes

  feng-j678

  committed
  83f7d17

Commits on Jun 2, 2023

• add logic for quote count after truncation

  feng-j678

  committed
  ece5ad6

• modify the quote_count logic and set_package_assessment_status to use map for performance optimization

  feng-j678

  committed
  f213d6f

• fix file formating and add more tests for report_sequence_number_changed_termination

  feng-j678

  committed
  7114d6d

• add more unit test

  feng-j678

  committed
  8242685

• remove unused variables in test_status

  feng-j678

  committed
  faa0bbc

Commits on Jun 5, 2023

• add truncation logic for installation

  feng-j678

  committed
  e9800ab

• modify the set_package_install_status and classification to use map for performance issue

  feng-j678

  committed
  fdf7d8f

Commits on Jun 6, 2023

• add unit tests for installation truncation

  feng-j678

  committed
  1555c07

• add more unit test for testing installation truncation

  feng-j678

  committed
  45823ac

• reset installation_tmp_map

  feng-j678

  committed
  3e1cd68

• add classification to test_installatio_keep_5

  feng-j678

  committed
  6d6f8fa

Commits on Jun 7, 2023

• revert set_pacakge_install_status to double forloops because values in map in 2.7 are randomize

  feng-j678

  committed
  89e99da

• change hashmap to orderedmap because existing utest check for order of elements

  feng-j678

  committed
  beb92bb

• add unit test for get latest complete status file

  feng-j678

  committed
  9ea154e

• add test for get latest complete status file

  feng-j678

  committed
  e0d19b6

• add descriptions to truncation tests

  feng-j678

  committed
  790bc13

• add unit test for ordered map

  feng-j678

  committed
  cfec898

Commits on Jun 8, 2023

• add test for checking if complete status is dir

  feng-j678

  committed
  274d69b

• add unit test for orderdict()

  feng-j678

  committed
  7fd4383

• Modify Agent status Constants and revert white spaces

  feng-j678

  committed
  eb353e6

• this has assessment truncatin logic bad format

  feng-j678

  committed
  0577a4d

• revert status handler format to as master version

  feng-j678

  committed
  a5c82fa

• revert code format from master version

  feng-j678

  committed
  fcc738e

• modify variable naming convention add unit for latest complete status

  feng-j678

  committed
  662b176

• change map to use orderdict to preserve element ordering because of py27

  feng-j678

  committed
  65ee40d

• add unit test for ordered map

  feng-j678

  committed
  65cf245

Commits on Jun 12, 2023

• chnage test_value to patch_count_for_test

  feng-j678

  committed
  3d186b9

Commits on Jun 13, 2023

• modify the tombstone according to classifications

  feng-j678

  committed
  4b60b38

• modify unit tests for new tombstone records

  feng-j678

  committed
  3d4b97f

• add unit test for complete status file is a dir

  feng-j678

  committed
  4782bc3

• add unit test for log_truncated_removed_packages_if_any

  feng-j678

  committed
  0178e9d

Commits on Jun 14, 2023

• merge new changes from assessment_truncation branch

  feng-j678

  committed
  668aed6

• merge in changes from assessment_truncation branch

  feng-j678

  committed
  3a8d421

Commits on Jun 18, 2023

• resolve unit tests failures

  feng-j678

  committed
  4f4456e

• resolved unit tests problems

  feng-j678

  committed
  3a27c9e

• failing unit tests

  feng-j678

  committed
  aed5b9a

• debug why unit tests are failing in coremain

  feng-j678

  committed
  5695532

• add runtime.stop() to fix failing

  feng-j678

  committed
  5683aa5

Commits on Jun 19, 2023

• fix unit test by correcting message patch tombstone record

  feng-j678

  committed
  d5ee671

• modify unit test to whether file size is > or < agent file size limit

  feng-j678

  committed
  d41b75b

• to reuse __set_errors_json for truncation

  feng-j678

  committed
  64aafd1

• remove code parameter in recompose logic

  feng-j678

  committed
  25ee9b1

• remove checks for agent file size limit in unit test per summary patches

  feng-j678

  committed
  8b5c729

Commits on Jun 20, 2023

• add get_index method for low pri

  feng-j678

  committed
  8ba655b

Commits on Jun 22, 2023

• add logic for high and low pri intallation

  feng-j678

  committed
  3b27791

• add method description to __get_installation_packages_index

  feng-j678

  committed
  7b8246b

• extract assessment tombstone into a separate method

  feng-j678

  committed
  510a6da

• refactor truncation main logic

  feng-j678

  committed
  02fd327

• merge changes from master to fix ci.yml and update statushandler

  feng-j678

  committed
  8bbb522

• add code flow to write_status_file()

  feng-j678

  committed
  67576b3

Commits on Jun 23, 2023

• refactor assessment cod, remove quote_count and use fix escape byte

  feng-j678

  committed
  dfcc478

• Merge branch 'master' of https://github.com/Azure/LinuxPatchExtension into user/johnfeng/add_installation_truncation

  feng-j678

  committed
  88b6137

• refactor __recompose_truncated_status_file

  feng-j678

  committed
  57e1cf5

• fix code to consider no assessment truncation, everything remains same

  feng-j678

  committed
  141e6c1

Commits on Jun 25, 2023

• update truncation exception message unit test

  feng-j678

  committed
  33fd85a

• remove unncessary removed_packages_details method

  feng-j678

  committed
  2c64887

• refactor truncation main logic

  feng-j678

  committed
  716d1d3

• add get status file w/o list method

  feng-j678

  committed
  9f600ed

• rename truncation varibles

  feng-j678

  committed
  e4f682b

Commits on Jun 26, 2023

• use while loop to checking size limit

  feng-j678

  committed
  4a559bd

• test fail pipeline

  feng-j678

  committed
  3023038

• add low_index_pri logic, status unit tests passed

  feng-j678

  committed
  0c0d8da

• add low_pri index logic, pass all tests

  feng-j678

  committed
  8f45390

Commits on Jun 27, 2023

• update unit tests and code flow

  feng-j678

  committed
  f4a99c0

• add extra to bottom of status handler.py

  feng-j678

  committed
  0e8650b

Commits on Jul 5, 2023

• change orderdict to collections.ordereddict

  feng-j678

  committed
  a098abc

Commits on Jul 6, 2023

• change get latest complete status file to remove older complete status

  feng-j678

  committed
  e474c34

• change forward slash to back slash in glob.glob:

  feng-j678

  committed
  89b88ef

• remove unused imports

  feng-j678

  committed
  6301415

• merge in changes from complete_status_file_logic branch

  feng-j678

  committed
  6342db0

Commits on Jul 26, 2023

• pr review with todos and comments

  feng-j678

  committed
  9004e51

• commit 1 -refactor to resuse substatus_message method

  feng-j678

  committed
  1aa276b

• commit 2 - rename variables

  feng-j678

  committed
  e65e8b4

• commit 3 - refactor global varibles assessment, installation packages copy

  feng-j678

  committed
  21f6394

• Merge branch 'master' of https://github.com/Azure/LinuxPatchExtension into user/johnfeng/add_installation_truncation

  feng-j678

  committed
  9e428d6

Commits on Jul 27, 2023

• change size_limit to capacity

  feng-j678

  committed
  e798676

• commit 2 combine assessment and installation logic into one

  feng-j678

  committed
  997c758

• change keep latest complete status logic to keep 10

  feng-j678

  committed
  7b539a1

• remove the break logic from current_list - diff

  feng-j678

  committed
  97727f1

• chnage test_assessment_operation_success_truncation_over_capacity because list in py27 isnt by order

  feng-j678

  committed
  f363707

• fix __removed_older_complete_status_files bug

  feng-j678

  committed
  f90b2f8

Commits on Jul 28, 2023

• commit 1 - refactor var, method, and combin logic in __apply_truncation_process

  feng-j678

  committed
  2c99f29

• commit 2 add fake version to tombstone, modify __removed_older_complete_status_files

  feng-j678

  committed
  165ef8e

• commit 3 update pesudo code

  feng-j678

  committed
  3d7ca29

• commit 4 remove assertion on complete status deletion becuase github runner creates tmp files randomly

  feng-j678

  committed
  e6ae224

Commits on Aug 4, 2023

• refactor truncation logic to write_complete_status_file()

  feng-j678

  committed
  6a8b0f1

• commit 2 refactor substatus_msg

  feng-j678

  committed
  7cf753f

Commits on Aug 5, 2023

• commit 3 add write status comment

  feng-j678

  committed
  75afd05

Commits on Aug 7, 2023

• commi t 4 refactor assessment installation variables names

  feng-j678

  committed
  e8305a5

• Merge branch 'master' of https://github.com/Azure/LinuxPatchExtension into user/johnfeng/add_installation_truncation

  feng-j678

  committed
  a89510d

• commit 1 move the max_list_capacity - keepfive logic to __apply_truncation_process

  feng-j678

  committed
  df93219

• commit 2 add utf-8 encode to unit test

  feng-j678

  committed
  0623c2f

• commit 3 change variables in __get_new_size_in_bytes_after_truncation

  feng-j678

  committed
  6e55396

• commit 4 simplify recompose assessment and installation substatus msg() and add unit tests

  feng-j678

  committed
  f8d5e46

• add extra space to eof

  feng-j678

  committed
  cd87ebc

Commits on Aug 9, 2023

• commit 1 change the while loop condition to status_file_byte_size > 126kb and update unit tests name

  feng-j678

  committed
  30c05d3

• commit 2 update pesuedo code

  feng-j678

  committed
  9245c3b

• commit 3 refactor methods decriptions

  feng-j678

  committed
  22d6d65

• test self.assertEqual(len(message_patches), 6) is 6 not 7

  feng-j678

  committed
  ba077d2

• change self.assertTrue(len(message_patches) >= 6)

  feng-j678

  committed
  5fc2dd9

• add performance for truncation

  feng-j678

  committed
  28d81d7

• update test_installation_truncation_with_only_install_packages_over_size_limit

  feng-j678

  committed
  a328491

• correct end.time - start.time in unit test

  feng-j678

  committed
  1b52a78

Commits on Aug 10, 2023

• remove comments and refactor __update_substatus_msg_packages_and_errors

  feng-j678

  committed
  71fdd6e

Commits on Aug 14, 2023

• commit 1 refactor variables, methods

  feng-j678

  committed
  ac0db9b

• commit 2 add named parameters

  feng-j678

  committed
  200e37d

Commits on Aug 18, 2023

• refactor code

  feng-j678

  committed
  40eac52

• refactor __recompose_truncated_status_file

  feng-j678

  committed
  cd30132

Commits on Aug 21, 2023

• merge in changes from master

  feng-j678

  committed
  9d083f8

Commits on Aug 29, 2023

• reset code changes because branch is ahead of master, refactor variable/method names

  feng-j678

  committed
  f34c848

• add unit test for validating the log message

  feng-j678

  committed
  63351e2

• change write log to tmp and read from it

  feng-j678

  committed
  d775cea

• move write to temp log file into a method for unit test

  feng-j678

  committed
  c2fdc5b

• refact substatus_msg_at_index to substatus_message

  feng-j678

  committed
  442593c

Commits on Sep 11, 2023

• add FORCE_WRITE_TRUNCATION constant, add check for 1 min timestamp conditional checker, and unit test

  feng-j678

  committed
  aa7fb97

• add unit test for 1 min delay

  feng-j678

  committed
  7120636

Commits on Sep 28, 2023

• add checker for delaying truncation logic and unit test

  feng-j678

  committed
  764b59b

• Merge branch 'master' of https://github.com/Azure/LinuxPatchExtension into user/johnfeng/add_installation_truncation

  feng-j678

  committed
  cf4b555

Commits on Sep 29, 2023

• add new comments and reset truncation boolean in load_status_file_component

  feng-j678

  committed
  51a3d53

Commits on Oct 3, 2023

• refactor method names

  feng-j678

  committed
  6a92ec0

Commits on Oct 19, 2023

• initialize truncation_timestamp inside __check_truncation_time_passed_by_x_sec

  feng-j678

  committed
  831cb70

• move self.__is_file_truncated logic to __check_status_file_size_perform_truncation_if_needed

  feng-j678

  committed
  2521e8a

Commits on Oct 20, 2023

• extract tombstone logic into a variable

  feng-j678

  committed
  d322898

• rename var and method, and set Constants.StatusTruncationConfig.NO_TRUNCATION_IN_X_SEC = -1

  feng-j678

  committed
  2d411e1

• change NO_TRUNCATION_IN_X_SEC = -1 to ensure truncation happens

  feng-j678

  committed
  424953d