Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[OIL] Split the API reference markdown into smaller files and use templates to generate it. #5338

Merged
merged 2 commits into from
Mar 11, 2024
Merged

[OIL] Split the API reference markdown into smaller files and use templates to generate it. #5338

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
8ec322a
Split the API reference markdown into smaller files and use templates…
kc284 Dec 20, 2023
37821d8
Corrections as per code review.
kc284 Mar 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,15 +74,20 @@ schema:
dune runtest ocaml/idl

doc:
dune build --profile=$(PROFILE) ocaml/idl/datamodel_main.exe
#html
dune build --profile=$(PROFILE) -f @ocaml/doc/jsapigen
mkdir -p $(XAPIDOC)/html
cp -r _build/default/ocaml/doc/api $(XAPIDOC)/html
cp _build/default/ocaml/doc/branding.js $(XAPIDOC)/html
cp ocaml/doc/*.js ocaml/doc/*.html ocaml/doc/*.css $(XAPIDOC)/html
dune exec --profile=$(PROFILE) -- ocaml/idl/datamodel_main.exe -closed -markdown $(XAPIDOC)/markdown
cp ocaml/doc/*.dot ocaml/doc/doc-convert.sh $(XAPIDOC)
#markdown
dune build --profile=$(PROFILE) -f @ocaml/idl/markdowngen
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we could define some install sections in dune and let dune install things into the right places. The method of installing files in the Makefile was from the time when we didn't have dune.
Then also the version substitutions inside the markdown files might work

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See https://dune.readthedocs.io/en/stable/dune-files.html#install

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unless I'm missing something, it doesn't seem to me that I can do this efficiently with the version of dune we're currently using. All the useful stanzas (source_trees, glob_files, include) have been added in various 3.x versions.The install sections I see in other dune files in the repo handle single files, but here there is a good number of autogenerated files that have to be installed.

mkdir -p $(XAPIDOC)/markdown
cp -r _build/default/ocaml/idl/autogen/*.md $(XAPIDOC)/markdown
cp -r _build/default/ocaml/idl/autogen/*.yml $(XAPIDOC)/markdown
find ocaml/doc -name "*.md" -not -name "README.md" -exec cp {} $(XAPIDOC)/markdown/ \;
#other
cp ocaml/doc/*.dot ocaml/doc/doc-convert.sh $(XAPIDOC)
psafont marked this conversation as resolved.
Show resolved Hide resolved
# Build manpages, networkd generated these
dune build --profile=$(PROFILE) -f @man

Expand Down
8 changes: 6 additions & 2 deletions ocaml/doc/basics.md
View file
Open in desktop
kc284 marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
@@ -1,12 +1,16 @@
---
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These headers are a particularity of XS docs. If there is an issue with having them here, I can remove them and have the pipeline insert them later (since it will have to replace the @root@ placeholders in some files anyway).

layout: doc
---

# API Basics

This document defines the XenServer Management API - an interface for remotely
configuring and controlling virtualised guests running on a Xen-enabled host.
configuring and controlling virtualized guests running on a Xen-enabled host.

The API is presented here as a set of Remote Procedure Calls (RPCs). There are
two supported wire formats, one based upon [XML-RPC](http://xmlrpc.scripting.com/spec.html)
and one based upon [JSON-RPC](http://www.jsonrpc.org) (v1.0 and v2.0 are both
recognised). No specific language bindings are prescribed, although examples
recognized). No specific language bindings are prescribed, although examples
will be given in the python programming language.

Although we adopt some terminology from object-oriented programming,
Expand Down
4 changes: 4 additions & 0 deletions ocaml/doc/vm-lifecycle.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
layout: doc
---

# VM Lifecycle

The following diagram shows the states that a VM can be in
Expand Down
14 changes: 9 additions & 5 deletions ocaml/doc/wire-protocol.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
layout: doc
---

# Wire Protocol for Remote API Calls

API calls are sent over a network to a Xen-enabled host using an RPC protocol.
Expand Down Expand Up @@ -178,7 +182,7 @@ following manner:

* our `void` type is transmitted as an empty string.

Both versions 1.0 and 2.0 of the JSON-RPC wire format are recognised and,
Both versions 1.0 and 2.0 of the JSON-RPC wire format are recognized and,
depending on your client library, you can use either of them.

### JSON-RPC v1.0
Expand Down Expand Up @@ -486,7 +490,7 @@ session reference is returned under the key `Value` in the resulting dictionary
... "version", "originator")['Value']
```

This is what the call looks like when serialised
This is what the call looks like when serialized

```xml
<?xml version='1.0'?>
Expand Down Expand Up @@ -530,7 +534,7 @@ Once a reference to a VM has been acquired, a lifecycle operation may be invoked
In this case the `start` message has been rejected, because the VM is
a template, and so an error response has been returned. These high-level
errors are returned as structured data (rather than as XML-RPC faults),
allowing them to be internationalised.
allowing them to be internationalized.

Rather than querying fields individually, whole _records_ may be returned at once.
To retrieve the record of a single object as a python dictionary:
Expand Down Expand Up @@ -575,7 +579,7 @@ reference:
... "user", "passwd", "version", "originator")
```

`pyjsonrpc` uses the JSON-RPC protocol v2.0, so this is what the serialised
`pyjsonrpc` uses the JSON-RPC protocol v2.0, so this is what the serialized
request looks like:

```json
Expand Down Expand Up @@ -623,7 +627,7 @@ Once a reference to a VM has been acquired, a lifecycle operation may be invoked

In this case the `start` message has been rejected because the VM is
a template, hence an error response has been returned. These high-level
errors are returned as structured data, allowing them to be internationalised.
errors are returned as structured data, allowing them to be internationalized.

Rather than querying fields individually, whole _records_ may be returned at once.
To retrieve the record of a single object as a python dictionary:
Expand Down
12 changes: 12 additions & 0 deletions ocaml/idl/autogen/api-ref-autogen.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
---
layout: doc
---

# API Reference

Version **@xapi-version@**
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there an easy way to have this replaced during build time? If not, I can have the pipeline replace it later).

Copy link
Member

@psafont psafont Jan 9, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently, the version is embedded into binaries using the Xapi_version module and using the dune-build-info library. One way to do it would be to it would be to create an ocaml binary that processes this file.

One way you can try is to use dune subst, and use %%VERSION%%. I believe it's intended to be used project-wide, but you're welcome to try

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the tip; I won't consider this as a blocker for this PR though (it's probably not a big change, but I only have time to work on this on and off, so I'd rather get the PR in the soonest possible and improve the versioning later).

Copy link
Member

@psafont psafont Jan 12, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Had a look, and the binary that generates the files already has access to this library, So it can already use mustache templates to generate it.

The only complication is that the binary needs to be "installed" before it generated the files so it can pick up the version, because of how dune injects the version, I've done this in psafont@13da274

But it needs testing to ensure that the version injected in the CI is simply the tag, and not either of these types:

# API Reference

Version 23.32.0-52-g13da274

# API Reference

Version 24.999.999-dirty

(this happens when the commit is untagged, and when make is run before make doc, respectively)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed, building in the CI yields 0.0.dev. In the interest of avoiding taking up too much of people's time, I can always leave it as is and replace it later in the pipeline alongside @root@.


- [Classes](@root@management-api/classes.html)
- [Relationships Between Classes](@root@management-api/relationships-between-classes.html)
- [Types](@root@management-api/types.html)
- [ErrorHandling](@root@management-api/api-ref-autogen-errors.html)
6 changes: 6 additions & 0 deletions ocaml/idl/autogen/dune
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
(alias
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is an alias with a name and dependency, but where is the action?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No action, just using source_tree to include all the files that will be generated in here. We have a few more dune files with rules like this.

(name markdowngen)
(deps
(source_tree .)
)
)
19 changes: 12 additions & 7 deletions ocaml/idl/datamodel.ml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2057,7 +2057,10 @@ module Bond = struct
let t =
create_obj ~in_db:true ~in_product_since:rel_miami ~in_oss_since:None
~persist:PersistEverything ~gen_constructor_destructor:false ~name:_bond
~descr:"" ~gen_events:true ~doccomments:[]
~descr:
"A Network bond that combines physical network interfaces, also known \
as link aggregation"
~gen_events:true ~doccomments:[]
~messages_default_allowed_roles:_R_POOL_OP ~doc_tags:[Networking]
~messages:[create; destroy; set_mode; set_property]
~contents:
Expand Down Expand Up @@ -5985,7 +5988,7 @@ module DR_task = struct
)
; (Set String, "whitelist", "The devices to use for disaster recovery")
]
~result:(Ref _dr_task, "The reference to the created task")
~result:(Ref _dr_task, "The reference of the created DR_task")
~doc:
"Create a disaster recovery task which will query the supplied list of \
devices"
Expand Down Expand Up @@ -6202,7 +6205,7 @@ module Blob = struct
}
]
~doc:"Create a placeholder for a binary blob" ~flags:[`Session]
~result:(Ref _blob, "The reference to the created blob")
~result:(Ref _blob, "The reference of the created blob")
~allowed_roles:_R_POOL_OP ()

let destroy =
Expand Down Expand Up @@ -6889,7 +6892,8 @@ module GPU_group = struct
; param_default= Some (VMap [])
}
]
~result:(Ref _gpu_group, "") ~allowed_roles:_R_POOL_OP ()
~result:(Ref _gpu_group, "The reference of the created GPU_group")
~allowed_roles:_R_POOL_OP ()

let destroy =
call ~name:"destroy"
Expand Down Expand Up @@ -7041,7 +7045,7 @@ module VGPU = struct
; param_default= Some (VRef null_ref)
}
]
~result:(Ref _vgpu, "reference to the newly created object")
~result:(Ref _vgpu, "The reference of the created VGPU object")
~allowed_roles:_R_POOL_OP ()

let destroy =
Expand Down Expand Up @@ -7356,7 +7360,7 @@ module PVS_proxy = struct

let create =
call ~name:"create" ~doc:"Configure a VM/VIF to use a PVS proxy"
~result:(Ref _pvs_proxy, "the new PVS proxy")
~result:(Ref _pvs_proxy, "The reference of the created PVS proxy")
~params:
[
(Ref _pvs_site, "site", "PVS site that we proxy for")
Expand Down Expand Up @@ -7626,7 +7630,8 @@ module USB_group = struct
; param_default= Some (VMap [])
}
]
~result:(Ref _usb_group, "") ~allowed_roles:_R_POOL_ADMIN ()
~result:(Ref _usb_group, "The reference of the created USB_group")
~allowed_roles:_R_POOL_ADMIN ()

let destroy =
call ~name:"destroy" ~lifecycle
Expand Down
3 changes: 2 additions & 1 deletion ocaml/idl/datamodel_certificate.ml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,8 @@ let certificate_type =
)

let t =
create_obj ~name:_certificate ~descr:"Description" ~doccomments:[]
create_obj ~name:_certificate
~descr:"An X509 certificate used for TLS connections" ~doccomments:[]
~gen_constructor_destructor:false ~gen_events:true ~in_db:true ~lifecycle
~persist:PersistEverything ~in_oss_since:None
~messages_default_allowed_roles:_R_READ_ONLY
Expand Down
2 changes: 1 addition & 1 deletion ocaml/idl/datamodel_errors.ml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1455,7 +1455,7 @@ let _ =
~doc:
"The requested update could not be found. Please upload the update \
again. This can occur when you run xe update-pool-clean before xe \
update-apply. "
update-apply."
() ;
error Api_errors.update_pool_apply_failed ["hosts"]
~doc:"The update cannot be applied for the following host(s)." () ;
Expand Down
2 changes: 1 addition & 1 deletion ocaml/idl/datamodel_host.ml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1369,7 +1369,7 @@ let set_power_on_mode =
; (Changed, rel_stockholm, "Removed iLO script")
]
~in_product_since:rel_midnight_ride
~doc:"Set the power-on-mode, host, user and password "
~doc:"Set the power-on-mode, host, user and password"
~params:
[
(Ref _host, "self", "The host")
Expand Down
2 changes: 1 addition & 1 deletion ocaml/idl/datamodel_main.ml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@ let _ =
in

if !markdown_mode then
Markdown_backend.all api !dirname ;
Markdown_backend.all api ;

if !dirname <> "" then Unix.chdir !dirname ;
if !dot_mode then
Expand Down
11 changes: 11 additions & 0 deletions ocaml/idl/dune
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,13 +30,24 @@
(modules datamodel_main dot_backend dtd_backend markdown_backend)
(libraries
dune-build-info
mustache
xapi-datamodel
xapi-stdext-std
xapi-stdext-pervasives
xapi-stdext-unix
)
)

(rule
(alias markdowngen)
(deps
(:x datamodel_main.exe)
(source_tree templates)
)
(package xapi-datamodel)
(action (run %{x} -closed -markdown))
)

(test
(name schematest)
(modes exe)
Expand Down
Loading
Loading