Data nodes sorted #2128

lePici · 2023-11-09T13:01:46Z

Overview of changes

Plugin type 'lyds_tree' is integrated. Used in metadata and it is
internal, which is distinguished by the unset print callback.
Internal metadata must not be printed.
Leaf-list and list data instances are now automatically sorted.
Both the explicit and implicit YANG statement 'ordered-by system'
will cause the nodes to be ordered.
The first instance of the list or leaf-list of node contain
new metadata named 'lyds_tree'.
Data nodes are sorted only if their data types have
callback 'sort' set.
Sorting can be turned off so far only by adding the 'ordered-by user'
statement to the YANG model.
If the sort fails for some reason, the node is still inserted
as the last instance. A warning message informs about this situation.
The time required for sorting should be relatively small thanks to
the Red-black tree implementation.
Memory requirements will now increase by 40 bytes per list/leaf-list
data node, plus metadata structure overhead.

perf

memory

test: 'create new text' without leaf-list inserts

1000 nodes
devel - 3,177,195
lyds_tree - 3,220,116
difference: 42,921

100000 nodes
devel - 122,591,019
lyds_tree - 126,593,940
difference: 4,002,921

40 bytes per lyd_node + metadata overhead

branch lyds_tree

1000-10

ly_perf:                                                                                                                                         
        data set size: 1000                                                                                                                      
        each test executed: 10 times                                                                                                             
                                                                                                                                                 
| create new text .................. | 0.002257 s |                                                                                              
| create new bin ................... | 0.001277 s |                                                                                              
| create path ...................... | 0.004225 s |                                                                                              
| validate ......................... | 0.000314 s |                                                                                              
| parse xml mem validate ........... | 0.003583 s |                                                                                              
| parse xml mem no validate ........ | 0.002826 s |                                                                                              
| parse xml file no validate format  | 0.002939 s |                                                                                              
| parse json mem validate .......... | 0.002679 s |                                                                                              
| parse json mem no validate ....... | 0.001798 s |                                                                                              
| parse json file no validate format | 0.001958 s |                                                                                              
| parse lyb mem validate ........... | 0.002584 s |                                                                                              
| parse lyb mem no validate ........ | 0.001783 s |                                                                                              
| parse lyb file no validate ....... | 0.001830 s |                                                                                              
| print xml ........................ | 0.001235 s |                                                                                              
| print json ....................... | 0.000796 s |                                                                                              
| print lyb ........................ | 0.000626 s |                                                                                              
| dup .............................. | 0.000496 s |                                                                                              
| free ............................. | 0.000513 s |                                                                                              
| xpath find ....................... | 0.000731 s |                                                                                              
| xpath find hash .................. | 0.000005 s |                                                                                              
| compare same ..................... | 0.000134 s |                                                                                              
| diff same ........................ | 0.001968 s |                                                                                              
| diff no same ..................... | 0.002004 s |                                                                                              
| merge same ....................... | 0.000978 s |                                                                                              
| merge no same .................... | 0.001157 s |                                                                                              
| merge no same destruct ........... | 0.000640 s |

100000-3

ly_perf:                                                                                                                                         
        data set size: 100000                                                                                                                    
        each test executed: 3 times                                                                                                              
                                                                                                                                                 
| create new text .................. | 0.249784 s |                                                                                              
| create new bin ................... | 0.158706 s |                                                                                              
| create path ...................... | 0.447686 s |                                                                                              
| validate ......................... | 0.049997 s |                                                                                              
| parse xml mem validate ........... | 0.459106 s |                                                                                              
| parse xml mem no validate ........ | 0.328475 s |                                                                                              
| parse xml file no validate format  | 0.342890 s |                                                                                              
| parse json mem validate .......... | 0.348044 s |                                                                                              
| parse json mem no validate ....... | 0.227186 s |                                                                                              
| parse json file no validate format | 0.244726 s |                                                                                              
| parse lyb mem validate ........... | 0.338318 s |                                                                                              
| parse lyb mem no validate ........ | 0.215666 s |                                                                                              
| parse lyb file no validate ....... | 0.215394 s |                                                                                              
| print xml ........................ | 0.119830 s |                                                                                              
| print json ....................... | 0.090757 s |                                                                                              
| print lyb ........................ | 0.053704 s |                                                                                              
| dup .............................. | 0.097857 s |                                                                                              
| free ............................. | 0.104160 s |                                                                                              
| xpath find ....................... | 0.085618 s |                                                                                              
| xpath find hash .................. | 0.000011 s |                                                                                              
| compare same ..................... | 0.027562 s |                                                                                              
| diff same ........................ | 0.309946 s |                                                                                              
| diff no same ..................... | 0.329439 s |                                                                                              
| merge same ....................... | 0.159534 s |                                                                                              
| merge no same .................... | 0.225557 s |                                                                                              
| merge no same destruct ........... | 0.110938 s |

branch devel

1000-10

ly_perf:                                                                                                                                         
        data set size: 1000
        each test executed: 10 times

| create new text .................. | 0.002367 s |
| create new bin ................... | 0.001393 s |
| create path ...................... | 0.004338 s |
| validate ......................... | 0.000333 s |
| parse xml mem validate ........... | 0.003538 s |
| parse xml mem no validate ........ | 0.002913 s |
| parse xml file no validate format  | 0.003118 s |
| parse json mem validate .......... | 0.002562 s |
| parse json mem no validate ....... | 0.001946 s |
| parse json file no validate format | 0.002321 s |
| parse lyb mem validate ........... | 0.002511 s |
| parse lyb mem no validate ........ | 0.001914 s |
| parse lyb file no validate ....... | 0.001943 s |
| print xml ........................ | 0.001435 s |
| print json ....................... | 0.001082 s |
| print lyb ........................ | 0.000596 s |
| dup .............................. | 0.000627 s |
| free ............................. | 0.000484 s |
| xpath find ....................... | 0.000865 s |
| xpath find hash .................. | 0.000007 s |
| compare same ..................... | 0.000148 s |
| diff same ........................ | 0.002134 s |
| diff no same ..................... | 0.002258 s |
| merge same ....................... | 0.001115 s |
| merge no same .................... | 0.001104 s |
| merge no same destruct ........... | 0.000554 s |

100000-3

ly_perf:                                                                                                                                         
        data set size: 100000                                                                                                                    
        each test executed: 3 times                                                                                                              
                                                                                                                                                 
| create new text .................. | 0.203631 s |                                                                                              
| create new bin ................... | 0.130454 s |                                                                                              
| create path ...................... | 0.411399 s |                                                                                              
| validate ......................... | 0.049272 s |                                                                                              
| parse xml mem validate ........... | 0.413775 s |                                                                                              
| parse xml mem no validate ........ | 0.340388 s |                                                                                              
| parse xml file no validate format  | 0.349116 s |                                                                                              
| parse json mem validate .......... | 0.318907 s |                                                                                              
| parse json mem no validate ....... | 0.238310 s |                                                                                              
| parse json file no validate format | 0.257953 s |                                                                                              
| parse lyb mem validate ........... | 0.302686 s |                                                                                              
| parse lyb mem no validate ........ | 0.232348 s |                                                                                              
| parse lyb file no validate ....... | 0.230928 s |                                                                                              
| print xml ........................ | 0.122265 s |                                                                                              
| print json ....................... | 0.088346 s |                                                                                              
| print lyb ........................ | 0.054410 s |                                                                                              
| dup .............................. | 0.103008 s |                                                                                              
| free ............................. | 0.077809 s |                                                                                              
| xpath find ....................... | 0.087260 s |                                                                                              
| xpath find hash .................. | 0.000013 s |                                                                                              
| compare same ..................... | 0.030518 s |                                                                                              
| diff same ........................ | 0.316554 s |                                                                                              
| diff no same ..................... | 0.329235 s |                                                                                              
| merge same ....................... | 0.164872 s |                                                                                              
| merge no same .................... | 0.201501 s |                                                                                              
| merge no same destruct ........... | 0.089045 s |

src/tree_data_sorted.c

src/tree_data_new.c

src/plugins_types/identityref.c

src/plugins_types/ipv4_address.c

src/plugins_types/ipv4_address_no_zone.c

src/plugins_types/ipv4_prefix.c

src/plugins_types/ipv6_prefix.c

src/plugins_types/lyds_tree.c

src/tree_data.c

src/tree_data_sorted.c

The lyd_insert_node() expecting first sibling.

... when deleting metadata in the lyd_parse_set_data_flags().

... from different contexts.

The function now has a new context parameter that may be needed when getting the canonical value. Type API version number has increased.

... implemented via new lyplg_type_sort_simple().

Simplification of sort and compare callback functions. The complexity of the comparison was needed in the old version of libyang.

The top parameter is removed.

1. Plugin type 'lyds_tree' is integrated. Used in metadata and it is internal, which is distinguished by the unset print callback. Internal metadata must not be printed. 2. Leaf-list and list data instances are now automatically sorted. 3. Both the explicit and implicit YANG statement 'ordered-by system' will cause the nodes to be ordered. 4. The first instance of the list or leaf-list of node contain new metadata named 'lyds_tree'. 5. Data nodes are sorted only if their data types have callback 'sort' set. 6. Sorting can be turned off so far only by adding the 'ordered-by user' statement to the YANG model. 7. If the sort fails for some reason, the node is still inserted as the last instance. A warning message informs about this situation. 8. The time required for sorting should be relatively small thanks to the Red-black tree implementation. 9. Memory requirements will now increase by 40 bytes per list/leaf-list data node, plus metadata structure overhead. 10. New internal lyds_tree plugin type.

1. When inserting, the last parameter can also be used for nodes with lyds_tree 2. The principle of lyds_tree is that it is created lazily. That is, if it is certain that the nodes are sorted, no metadata or lyds_tree is created. 3. If it is necessary to modify the sorted nodes, the lyds_tree is created additionally. This means that inserting the first node can lead to a delay. 4. The LYD_PARSE_ORDERED flag now has a positive performance effect because the lyds_tree is created lazily.

Unlink and then insert is inefficient for sorted (lyds) list and leaf-list nodes. Also improved lyd_unlink_siblings().

If the LYD_PARSE_ORDERED flag is set incorrectly, the data may be unordered, a warning will inform about this if the DEBUG variable is set. But as soon as the lyds tree is created later, the data will be sorted.

Added return value for functions lyd_unlink_siblings() and lyd_unlink_tree(), which causes ABI breaks.

There is no more freeing of memory and then allocation regarding regarding lyds data. Memory intended to be freed is temporarily not freed and instead waits for an opportunity to use it again.

This change avoids unnecessary finding of the first sibling in case the parent node is not available.

Function lyd_insert_node() has new options that ignore the ordering of data nodes. This is used in lyd_diff_add() because the diff as such does not need to be sorted.

Restored functionality regarding the sorting of opaq nodes and nodes with the LYD_EXT flag set.

Faster node insertion. There is no need to search for the 'anchor' for instances of the (leaf-)list, thus reducing the number of accesses to the hash table.

github-advanced-security bot found potential problems Nov 9, 2023

View reviewed changes

src/tree_data_sorted.c Fixed Show fixed Hide fixed

src/tree_data_new.c Fixed Show fixed Hide fixed

lePici force-pushed the lyds_tree branch from ededb53 to 693a583 Compare November 24, 2023 06:27

michalvasko requested changes Nov 24, 2023

View reviewed changes

lePici force-pushed the lyds_tree branch 2 times, most recently from 9333db4 to c824c0a Compare November 29, 2023 13:14

lePici force-pushed the lyds_tree branch from c824c0a to 86409b6 Compare January 11, 2024 17:20

lePici force-pushed the lyds_tree branch 2 times, most recently from 8026822 to 53e8c1d Compare February 9, 2024 14:48

michalvasko changed the base branch from devel to new_so February 12, 2024 08:02

michalvasko force-pushed the new_so branch from 27358f0 to 93f311f Compare February 12, 2024 08:06

lePici added 20 commits February 13, 2024 12:55

tree data REFACTOR bugfix doxygen documentation

ca16868

tree data BUGFIX in lyd_insert_sibling()

e7bd084

The lyd_insert_node() expecting first sibling.

parser BUGFIX references to siblings

38ea0fd

... when deleting metadata in the lyd_parse_set_data_flags().

tree schema BUGFIX LYS_ORDBY_MASK value

9eb9179

tree data REFACTOR added lyd_get_meta_annotation()

48a381e

data tree BUGFIX duplication of metadata

fcce169

... from different contexts.

tree data NEW lyd_unlink_meta_single()

2bb0173

tree data REFACTOR of _lyd_change_term()

49824dd

plugins types CHANGE compare function callback

3cc81e1

The function now has a new context parameter that may be needed when getting the canonical value. Type API version number has increased.

plugins types FEATURE sort callback in binary

86f23cf

plugins types FEATURE sort callback in bits

d05e392

plugins types FEATURE sort callback in boolean

024e76e

plugins types FEATURE sort callback in date_time

3af39c0

plugins types FEATURE sort callback in decimal64

52175a8

plugins types FEATURE sort callback in empty

796b167

... implemented via new lyplg_type_sort_simple().

plugins types FEATURE sort callback in enumeration

bf08ad5

plugins types FEATURE sort callback in hex_string

041e16a

plugins types FEATURE sort callback in identityref

d23e267

plugins types FEATURE sort callback in integer

1a92571

plugins types FEATURE sort callback in instanceid

3489a50

lePici added 15 commits February 13, 2024 12:55

plugins types REFACTOR for instanceid

7ea1bd0

Simplification of sort and compare callback functions. The complexity of the comparison was needed in the old version of libyang.

plugins types FEATURE sort callback in string

0e15ad2

plugins types FEATURE sort callback in union

96db3d6

plugins types FEATURE sort callback in inet_types

de4f12d

plugins types FEATURE sort callback in leafref

f4e5ffc

plugins types FEATURE sort callback in xpath

c1d2fa4

tree data REFACTOR lyd_free functions

b0af023

The top parameter is removed.

tree data REFACTOR of lyd_insert_node()

a229693

tree data OPTIMIZE unlink-insert for lyds nodes

0d90cdc

Unlink and then insert is inefficient for sorted (lyds) list and leaf-list nodes. Also improved lyd_unlink_siblings().

lyds tree OPTIMIZE data can be unordered

4642a04

If the LYD_PARSE_ORDERED flag is set incorrectly, the data may be unordered, a warning will inform about this if the DEBUG variable is set. But as soon as the lyds tree is created later, the data will be sorted.

tree data BUGFIX added return value for unlink

f15b1b8

Added return value for functions lyd_unlink_siblings() and lyd_unlink_tree(), which causes ABI breaks.

tree data OPTIMIZE lyds in lyd_merge()

a180411

There is no more freeing of memory and then allocation regarding regarding lyds data. Memory intended to be freed is temporarily not freed and instead waits for an opportunity to use it again.

tree data OPTIMIZE added first_sibling parameter

517273e

This change avoids unnecessary finding of the first sibling in case the parent node is not available.

lePici force-pushed the lyds_tree branch from 53e8c1d to 9dbf918 Compare February 13, 2024 12:12

lePici marked this pull request as ready for review February 13, 2024 12:13

lePici force-pushed the lyds_tree branch from 9dbf918 to 19771ee Compare February 13, 2024 14:31

lePici added 3 commits February 13, 2024 15:54

tree data OPTIMIZE diff and lyd_insert_node()

8880c3e

Function lyd_insert_node() has new options that ignore the ordering of data nodes. This is used in lyd_diff_add() because the diff as such does not need to be sorted.

tree data BUGFIX in ly_insert_node()

b5189da

Restored functionality regarding the sorting of opaq nodes and nodes with the LYD_EXT flag set.

tree data OPTIMIZE of lyd_dup() for (leaf-)list

e7ce2ab

Faster node insertion. There is no need to search for the 'anchor' for instances of the (leaf-)list, thus reducing the number of accesses to the hash table.

lePici force-pushed the lyds_tree branch from 19771ee to e7ce2ab Compare February 13, 2024 14:55

michalvasko approved these changes Feb 14, 2024

View reviewed changes

michalvasko merged commit 63fd1d1 into CESNET:new_so Feb 14, 2024
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Data nodes sorted #2128

Data nodes sorted #2128

lePici commented Nov 9, 2023 •

edited

Loading

Data nodes sorted #2128

Data nodes sorted #2128

Conversation

lePici commented Nov 9, 2023 • edited Loading

Overview of changes

perf

memory

branch lyds_tree

1000-10

100000-3

branch devel

1000-10

100000-3

lePici commented Nov 9, 2023 •

edited

Loading