Merge master into feature/perf #6078
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Before, having an escape sequence as part of the parameter would break grep: ``` $ xe vdi-list name-label=CentOS\ 7\ \(1grep: Unmatched ( or \( ``` Move back to pure bash processing for the prefix, since it's the weirdness of variable escaping leaving the bash context causing this. Signed-off-by: Andrii Sultanov <[email protected]>
Before, having an escape sequence as part of the parameter would break grep: ``` $ xe vdi-list name-label=CentOS\ 7\ \(1grep: Unmatched ( or \( ``` Move back to pure bash processing for the prefix, since it's the weirdness of variable escaping leaving the bash context causing this.
Signed-off-by: Guillaume <[email protected]>
Only exits script when an error occurs when getting the static IPv6 config info Signed-off-by: Benjamin Reis <[email protected]>
Dates converted from unix timestamps have subsecond precision, but dates serialized in the database do not. This is a change in behaviour introduced in 233b96b This fix was supposed to be included in the change, but got lost. No other users were found that hit this change in behaviour. Signed-off-by: Pau Ruiz Safont <[email protected]>
Only exits script when an error occurs when getting the static IPv6 config info
Dates converted from unix timestamps have subsecond precision, but dates serialized in the database do not. This is a change in behaviour introduced in 233b96b This fix was supposed to be included in the change, but got lost. No other users were found that hit this change in behaviour. Fixes the test about VMs pending guidances being removed when not needed, job 411209
For the context, we had an strange behavior where an host installed with IPv6 stack only got an Ipv4 address only during the first boot due to the start of a systemd service called chrony-wait. This is because during the first boot the **xcp-networkd** looks for a configuration from `/etc/firstboot.d/data/management.conf` and if the `MODE=none` it uses DHCP4. And as chrony-wait is started it got a source in IPv4 before XAPI had time to reconfigure it using IPv6 stack. So we think that if the MODE is none we shouldn't use DHCP4. Regards, Guillaume
Signed-off-by: xueqingz <[email protected]>
Bug fix for Go SDK, add an default Enum value to replace raising error for unknown Enum value. Build new SDK and samples, test pass XenRT job: 4100718.
Pending some final testing by QA, these commits have already been reviewed on the feature branch. This switches `select` calls to `Unixext.select` (which is implemented using `epoll`), and implements the few performance sensitive parts using epoll directly. It also enables some more tests with >1024 fds.
In pre-join check for pool.join, assert that there is no host pending mandatory guidance on the joining host or the pool coordinator. Signed-off-by: Gang Ji <[email protected]>
Defining as Copy of uri * uri results in JSON of `["Copy", ["uri", "uri"]]` but the unparse tries to enforce the `["uri", "uri"]` as a tuple, a type which is not supported by JSON. Signed-off-by: Mark Syms <[email protected]>
…l.in ./configure generates api_version.ml from api_version.ml.in, but their comments didn't match. Synchronize them, such that running ./configure doesn't result in committable changes. Also ignore the api_version.ml.in2 file that gets created. Signed-off-by: Edwin Török <[email protected]>
We cannot prevent dune from building them at the moment (`nodynlink` is disabled, and doesn't work with PIE executables). However disable installing all these files to avoid the .spec file complaining about additional cmxs files appearing when opam packages are reorganized. Signed-off-by: Edwin Török <[email protected]>
'update-dm-lifecycle' and 'python' only exists in a single directory,
tell 'dune' exactly where to look.
This speeds up incremental builds:
```
dune build @ocaml/xapi-storage/python/xapi/storage/api/v5/python --profile=release ran
3.41 ± 0.18 times faster than dune build @python --profile=release
dune build @ocaml/idl/update-dm-lifecycle -j 8 --profile=release ran
2.13 ± 0.19 times faster than dune build @update-dm-lifecycle -j 8 --profile=release --auto-promote
```
```
Benchmark 2: dune build @ocaml/xapi-storage/python/xapi/storage/api/v5/python --profile=release
Time (mean ± σ): 288.7 ms ± 3.7 ms [User: 229.5 ms, System: 58.9 ms]
Range (min … max): 282.5 ms … 293.5 ms 10 runs
Benchmark 2: dune build @ocaml/idl/update-dm-lifecycle -j 8 --profile=release
Time (mean ± σ): 581.4 ms ± 15.8 ms [User: 442.5 ms, System: 175.8 ms]
Range (min … max): 550.5 ms … 602.2 ms 10 runs
```
Signed-off-by: Edwin Török <[email protected]>
Use `opam-dune-lint`. This will make it easier to later merge opam packages and keep the merged package updated correctly. Although `opam-dune-lint` doesn't find all problems yet, see: ocurrent/opam-dune-lint#71 Signed-off-by: Edwin Török <[email protected]>
`set -x` is only needed when debugging the script. Don't flood 'make install' output. Signed-off-by: Edwin Török <[email protected]>
…ucceed Previously it failed because the symlink already existed. Signed-off-by: Edwin Török <[email protected]>
…re writing files When run inside Dune's sandbox you cannot rely on another rule having created the empty directories for you. Ensure that we create parent directories before we write a new file, otherwise we fail with a `Sys_error` about `No such file or directory`. Signed-off-by: Edwin Török <[email protected]>
Also add an 'alias generate' for the csharp/autogen/LICENSE file. All other sdk subdirs have a 'generate' alias in 'autogen', except for 'csharp', and without it 'make sdk' would fail to build due to the missing file. Signed-off-by: Edwin Török <[email protected]>
Avoid copying twice, get dune to install the files to the correct destination in one go. Also add a copy of XE_SR_ERRORCODES.xml, by default 'make install' would look for this in /opt/xensource, and writing there requires root. (This can be overriden with `./configure --share`). Since we are using `dune` to install the files now we need the file to always be present. Had to adjust the paths used by the CI. Uses dune directory targets, and the directory must be entirely under the control of these rules. There are some static files in autogen/ though, so move the generated ones to autogen-out, and then use 'cp -r' to copy over the static ones (there is no builtin dune action for the copy, there are individual copy actions, or a copy_files rule, but neither is suitable here) Signed-off-by: Edwin Török <[email protected]>
To avoid the .spec build complaining about newly installed files. These aren't used outside of XAPI anyway. Remove test on xenopsd.cmxs, which is no longer built (there is still a test on the simulator and main XAPI). Signed-off-by: Edwin Török <[email protected]>
…i-doc rule More efficient than install.sh. Also enables merging opam packages more easily, as it makes it clearer which package installs files where. 'xapi' and 'xe' are special cases because they install to /opt/xensource, so add a separate line with a different --prefix for them. We need a new xapi-debug package to install to /opt/xensource/debug which is a non-FHS compliant path, and we override its bindir to be /opt/xensource/debug. `vhd-tool` is special too, since it installs some files to /usr/libexec/xapi, but xapi itself is in /opt/xensource and its libexec would be /opt/xensource/libexec. Dune doesn't support installing to arbitrary directories, there is a 'misc' rule for that, but it is deprecated so I wouldn't start using it now. These rules generate multiple files in a directory, use dune's directory target support. Previously we would've copied the files multiple times. This is quite difficult to review, I suggest reviewing the outcome of the PR: On b123832 (do not installs cmxs commit): ``` rm /tmp/inst/1 -rf; make build install DESTDIR=/tmp/inst/1 && (cd /tmp/inst/1 && find ! -type d -printf '%p %M\n' ) >|ref ``` On the final commit of this PR: ``` rm /tmp/inst/2 -rf; make build install DESTDIR=/tmp/inst/2 && (cd /tmp/inst/2 && find ! -type d -printf '%p %M\n' ) >|this && sort -u ref >|refs && sort -u this >|thiss && diff -wu refs thiss ``` Signed-off-by: Edwin Török <[email protected]>
compare with: ``` rm /tmp/inst/2 -rf; make build install DESTDIR=/tmp/inst/2 && (cd /tmp/inst/2 && find ! -type d -printf '%p %M %8s\n' ) >|this && sort -u ref >|refs && sort -u this >|thiss && diff -wu refs thiss && echo OK ``` Signed-off-by: Edwin Török <[email protected]>
The dune build runs can't be parallelized, but the python and install.sh invocations and the `dune install` invocations can. Also remove the dependency between 'build' and 'install', if desired during development one can run `make build install`, however for the .spec build it is better to have a strict separation between build and install phases (this ensures that the install phase doesn't rebuild things by accident). Signed-off-by: Edwin Török <[email protected]>
…i-rrdd-plugin They use the same install destination. xapi-rrdd-plugin was never installed, so remove the public-name and opam package (but keep the sources, they might be useful as an example). TODO: opam dune lint Signed-off-by: Edwin Török <[email protected]>
These weren't being installed previously during a spec build, and had to be 'rm'-ed in the Makefile. Join them into a single package instead to reduce the number of opam files. Signed-off-by: Edwin Török <[email protected]>
Signed-off-by: Edwin Török <[email protected]>
Improves: 1bb4543 ("CP-51479: [maintenance]: install SDK files using dune rules") Signed-off-by: Edwin Török <[email protected]>
Signed-off-by: Rob Hoes <[email protected]>
The pvsproxy socket is available in both /opt/ and /run. Since /run is a more sensible location for a socket, use that one to allow the other to be removed in the future. Signed-off-by: Ross Lagerwall <[email protected]>
The function is now passed a plain fd rather than a Buf_io.t ("bio")
value, as it does not actually use the buffered channel and just read
from the fd directly (using `Http.read_http_request_header`).
There used to be an older version if `request_from_bio` that had an
option to read requests in a different way and that did use Buf_io. This
was called the "slow path" and was removed in bc2ff45 in favour of the
current "fast path". This further clean-up opportunity was missed at
that time.
Signed-off-by: Rob Hoes <[email protected]>
The function `check_reusable_inner` used Buf_io to read a fixed-length HTTP response and then discarded the buffer. This is functionally the same as using `Unixext.really_read_string`, so do that instead. Signed-off-by: Rob Hoes <[email protected]>
At this point, the only function in the entire code base that read from a Buf_io.t is `Http_svr.read_body` (apart from a test for Buf_io). However, it only does so if the buffer is not empty and falls back to reading directly from the fd is not. And since nothing else reads from a Buf_io, the buffer is always empty... Signed-off-by: Rob Hoes <[email protected]>
The main difference between the BufIO and FdIO cases was that the former calls `assert_credentials_ok` with the `callback` in its `~fn` parameter, while the latter executed the `callback` directly after the credentials check. The function `assert_credentials_ok` either calls `fn` or raises an exception. Well, nearly... It did not actually call `fn` in the unix socket case, where checks are bypassed. This looks unintended and this patch corrects it. This only affects the following handlers in xapi, which use BufIO and require RBAC checks: post_remote_db_access, post_remote_db_access_v2, get_wlb_report, get_wlb_diagnostics, get_audit_log. I guess those were simply never used on the unix socket. The other thing that happens when using `~fn` is that the function `Rbac_audit.allowed_post_fn_ok` is called after `~fn`. This writes an "ALLOWED OK" line to the audit log. I don't see a reason not to do the same in all cases. The outcome is that now both cases of `add_handler` do the same and only the channel types are different. In the following commit the two handler types are joining into a single one, which is now easier. Signed-off-by: Rob Hoes <[email protected]>
HTTP handlers of type BufIO did not actually read from through the
buffer at all. Instead, they all assert that the buffer is empty and
then simply use the file descriptor.
All HTTP handlers now directly use file descriptors. The handler type
simply becomes:
type 'a handler = Http.Request.t -> Unix.file_descr -> 'a -> unit
Signed-off-by: Rob Hoes <[email protected]>
Signed-off-by: Rob Hoes <[email protected]>
The pvsproxy socket is available in both /opt/ and /run. Since /run is a more sensible location for a socket, use that one to allow the other to be removed in the future.
If you install the XAPI RPMs in your koji build environment (e.g. to
build a package that depends on XAPI) then you couldn't build XAPI again
anymore because its unit tests were failing. They were failing because
they found some xapi hooks installed by the previous version of XAPI,
whereas normally there'd be none when the unit tests are running.
Disable running XAPI hooks during unit test, even if present we are not
expected to run them.
```
[exception] Unix.Unix_error(Unix.ENOENT, "connect", "")
Raised at Forkhelpers.execute_command_get_output_inner.(fun) in file "ocaml/forkexecd/lib/forkhelpers.ml", line 376, characters 10-19
Called from Xapi_stdext_pervasives__Pervasiveext.finally in file "ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml", line 24, characters 8-14
Re-raised at Xapi_stdext_pervasives__Pervasiveext.finally in file "ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml", line 39, characters 6-15
Called from Xapi_hooks.execute_hook.(fun) in file "ocaml/xapi/xapi_hooks.ml", line 77, characters 10-113
Called from Stdlib__Array.iter in file "array.ml", line 95, characters 31-48
Called from Xapi_host.destroy in file "ocaml/xapi/xapi_host.ml", line 1108, characters 2-98
Called from Dune__exe__Test_cluster_host.test_forget in file "ocaml/tests/test_cluster_host.ml", line 192, characters 2-42
Called from Alcotest_engine__Core.Make.protect_test.(fun) in file "src/alcotest-engine/core.ml", line 181, characters 17-23
Called from Alcotest_engine__Monad.Identity.catch in file "src/alcotest-engine/monad.ml", line 24, characters 31-35
```
Cleaning up the attic... It turned out that the `Buf_io` module, while used in the type signature of every HTTP handler, was not used for anything useful anymore. Step by step rationale in the commit messages, so best reviewed commit by commit.
When these metrics were collected internally, Xenctrl was queried every 5 seconds. After being split into plugins, they started querying domains (and other information) only on startup, so couldn't pick up new VMs and report their metrics without restarting. Signed-off-by: Andrii Sultanov <[email protected]>
Remove all the gating on cluster_health enabled as an experimental feature now that it is enabled by default. Signed-off-by: Vincent Liu <[email protected]>
Remove all the gating on cluster_health enabled as an experimental feature now that it is enabled by default.
#6067) When these metrics were collected internally, Xenctrl was queried every 5 seconds. After being split into plugins, they started querying domains (and other information) only on startup, so couldn't pick up new VMs and report their metrics without restarting.
SHA256 and SHA1 certificates' fingerprints do not get populated when the database is upgraded, so empty values need to be detected and amended on startup. Signed-off-by: Pau Ruiz Safont <[email protected]> Signed-off-by: Steven Woods <[email protected]>
This allows the CA certificate to be removed from the DB even if the certificate file does not exist. Signed-off-by: Steven Woods <[email protected]>
Without it, stats for bond's interfaces are not identified correctly. Fixes: bd4dda5 (IH-715 - rrdp-netdev: Remove double (de)serialization) Signed-off-by: Andrii Sultanov <[email protected]>
Reason: API errors in go sdk are generated from Api_errors.errors. Error is filled in Api_errors.errors when defined using add_error function. Error too_many_groups is not defined using add_error function. Fix: Define too_many_groups using add_error function. Signed-off-by: Changlei Li <[email protected]>
Reason: API errors in go sdk are generated from Api_errors.errors. Error is filled in Api_errors.errors when defined using add_error function. Error too_many_groups is not defined using add_error function. Fix: Define too_many_groups using add_error function.
psafont
approved these changes
Oct 24, 2024
These become warnings in ocaml 5.0+ Signed-off-by: Pau Ruiz Safont <[email protected]>
These become warnings in ocaml 5.0+
robhoes
approved these changes
Oct 24, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.