website

vincentarelbundock · Oct 20, 2023 · 21d92e4 · 21d92e4
1 parent 31361c0
commit 21d92e4
Show file tree

Hide file tree

Showing 99 changed files with 1,701 additions and 1,182 deletions.
diff --git a/book/articles/alternative_software.qmd b/book/articles/alternative_software.qmd
@@ -767,6 +767,57 @@ data.frame(bm$ContrastSummary)
 avg_slopes(ls.fe, dpar = "sigma", re_formula = NA)
 ```
 
+## `fmeffects`
+
+The [`fmeffects` package](https://cran.r-project.org/package=fmeffects) is described as follows:
+
+> fmeffects: Model-Agnostic Interpretations with Forward Marginal Effects. Create local, regional, and global explanations for any machine learning model with forward marginal effects. You provide a model and data, and 'fmeffects' computes feature effects. The package is based on the theory in: C. A. Scholbeck, G. Casalicchio, C. Molnar, B. Bischl, and C. Heumann (2022)
+
+As the name says, this package is focused on "forward marginal effects" in the context of machine learning models estimated using the `mlr3` or `tidymodels` frameworks. Since version 0.16.0, `marginaleffects` also supports these machine learning frameworks, and it covers a superset of the `fmeffects` functionality. Consider a random forest model trained on the `bikes` data:
+
+```{r, message = FALSE, warning = FALSE}
+library("mlr3verse")
+library("fmeffects")
+data("bikes", package = "fmeffects")
+task <- as_task_regr(x = bikes, id = "bikes", target = "count")
+forest <- lrn("regr.ranger")$train(task)
+```
+
+Now, we use the `avg_comparisons()` function to compute *centered* marginal effects:
+
+```{r}
+avg_comparisons(forest, variables = "temp", newdata = bikes)
+```
+
+We call this quantity "centered" because it represents the average effect of a change of 1 unit in `temp` about the observed value, that is, a change from 0.5 below to 0.5 above:
+
+```{r}
+lo <- transform(bikes, temp = temp - 0.5)
+hi <- transform(bikes, temp = temp + 0.5)
+mean(predict(forest, newdata = hi) - predict(forest, newdata = lo))
+```
+
+As described in the [`comparisons()` vignette](comparisons.html), it is easy to estimate "backward", "centered" or "forward" differences by supplying an appropriate function to the `variables` argument. For example, here is how to compute "forward" marginal effects:
+
+```{r}
+avg_comparisons(
+    forest,
+    variables = list("temp" = \(x) data.frame(x, x + 1)),
+    newdata = bikes)
+```
+
+This is equivalent to the key quantity reported by the `fmeffects` package:
+
+```{r}
+fmeffects::fme(
+    model = forest,
+    data = bikes,
+    target = "count",
+    feature = "temp",
+    step.size = 1)$ame
+```
+
+
 ## `effects`
 
 The [`effects` package](https://cran.r-project.org/package=effects) was created by John Fox and colleagues.

diff --git a/book/articles/machine_learning.qmd b/book/articles/machine_learning.qmd
@@ -131,28 +131,6 @@ avg_comparisons(
     newdata = bikes)
 ```
 
-## `fmeffects`: Forward vs. centered effects
-
-As the code in the `mlr3` section makes clear, the [`avg_comparisons()`](reference/comparisons.html) computes the effect of a "centered" change on the outcome. If we want to compute a "Forward Marginal Effect" instead, we can call:
-
-```{r}
-avg_comparisons(
-    forest,
-    variables = list("temp" = \(x) data.frame(x, x + 1)),
-    newdata = bikes)
-```
-
-This is equivalent to using the `fmeffects` package:
-
-```{r}
-fmeffects::fme(
-    model = forest,
-    data = bikes,
-    target = "count",
-    feature = "temp",
-    step.size = 1)$ame 
-```
-
 
 ## Partial Dependence Plots
 

diff --git a/docs/articles/NEWS.html b/docs/articles/NEWS.html
@@ -292,14 +292,14 @@
 </li>
           <li class="sidebar-item">
   <div class="sidebar-item-container"> 
-  <a href="../articles/machine_learning.html" class="sidebar-item-text sidebar-link">
- <span class="menu-text"><span class="chapter-number">20</span>&nbsp; <span class="chapter-title">Machine Learning</span></span></a>
+  <a href="../articles/matching.html" class="sidebar-item-text sidebar-link">
+ <span class="menu-text"><span class="chapter-number">20</span>&nbsp; <span class="chapter-title">Matching</span></span></a>
   </div>
 </li>
           <li class="sidebar-item">
   <div class="sidebar-item-container"> 
-  <a href="../articles/matching.html" class="sidebar-item-text sidebar-link">
- <span class="menu-text"><span class="chapter-number">21</span>&nbsp; <span class="chapter-title">Matching</span></span></a>
+  <a href="../articles/machine_learning.html" class="sidebar-item-text sidebar-link">
+ <span class="menu-text"><span class="chapter-number">21</span>&nbsp; <span class="chapter-title">Machine Learning</span></span></a>
   </div>
 </li>
           <li class="sidebar-item">

diff --git a/docs/articles/alternative_software.html b/docs/articles/alternative_software.html
diff --git a/docs/articles/bootstrap.html b/docs/articles/bootstrap.html
diff --git a/docs/articles/bootstrap_files/figure-html/unnamed-chunk-13-1.png b/docs/articles/bootstrap_files/figure-html/unnamed-chunk-13-1.png
diff --git a/docs/articles/brms.html b/docs/articles/brms.html
@@ -307,14 +307,14 @@
 </li>
           <li class="sidebar-item">
   <div class="sidebar-item-container"> 
-  <a href="../articles/machine_learning.html" class="sidebar-item-text sidebar-link">
- <span class="menu-text"><span class="chapter-number">20</span>&nbsp; <span class="chapter-title">Machine Learning</span></span></a>
+  <a href="../articles/matching.html" class="sidebar-item-text sidebar-link">
+ <span class="menu-text"><span class="chapter-number">20</span>&nbsp; <span class="chapter-title">Matching</span></span></a>
   </div>
 </li>
           <li class="sidebar-item">
   <div class="sidebar-item-container"> 
-  <a href="../articles/matching.html" class="sidebar-item-text sidebar-link">
- <span class="menu-text"><span class="chapter-number">21</span>&nbsp; <span class="chapter-title">Matching</span></span></a>
+  <a href="../articles/machine_learning.html" class="sidebar-item-text sidebar-link">
+ <span class="menu-text"><span class="chapter-number">21</span>&nbsp; <span class="chapter-title">Machine Learning</span></span></a>
   </div>
 </li>
           <li class="sidebar-item">
@@ -1084,8 +1084,8 @@ <h1 class="title">
 <span>    by <span class="op">=</span> <span class="st">"x"</span><span class="op">)</span></span>
 <span><span class="co">#&gt; </span></span>
 <span><span class="co">#&gt;  x Estimate 2.5 % 97.5 %</span></span>
-<span><span class="co">#&gt;  0    0.454 0.338  0.565</span></span>
-<span><span class="co">#&gt;  1    0.552 0.436  0.664</span></span>
+<span><span class="co">#&gt;  0    0.453 0.337  0.566</span></span>
+<span><span class="co">#&gt;  1    0.552 0.434  0.661</span></span>
 <span><span class="co">#&gt; </span></span>
 <span><span class="co">#&gt; Columns: x, estimate, conf.low, conf.high </span></span>
 <span><span class="co">#&gt; Type:  response</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
@@ -1099,7 +1099,7 @@ <h1 class="title">
 <span>    sample_new_levels <span class="op">=</span> <span class="st">"gaussian"</span><span class="op">)</span></span>
 <span><span class="co">#&gt; </span></span>
 <span><span class="co">#&gt;  Term Contrast Estimate  2.5 % 97.5 %</span></span>
-<span><span class="co">#&gt;     x    1 - 0   0.0965 0.0494  0.162</span></span>
+<span><span class="co">#&gt;     x    1 - 0   0.0959 0.0488  0.161</span></span>
 <span><span class="co">#&gt; </span></span>
 <span><span class="co">#&gt; Columns: term, contrast, estimate, conf.low, conf.high </span></span>
 <span><span class="co">#&gt; Type:  response</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
@@ -1116,8 +1116,8 @@ <h1 class="title">
 <span>  contrasts <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/cbind.html">cbind</a></span><span class="op">(</span><span class="st">"AME x"</span> <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op">(</span><span class="op">-</span><span class="fl">1</span>, <span class="fl">1</span><span class="op">)</span><span class="op">)</span>,</span>
 <span>  effects <span class="op">=</span> <span class="st">"integrateoutRE"</span><span class="op">)</span></span>
 <span><span class="va">bm</span><span class="op">$</span><span class="va">ContrastSummary</span> <span class="op">|&gt;</span> <span class="fu"><a href="https://rdrr.io/r/base/data.frame.html">data.frame</a></span><span class="op">(</span><span class="op">)</span></span>
-<span><span class="co">#&gt;            M        Mdn         LL        UL PercentROPE PercentMID   CI CIType ROPE  MID Label</span></span>
-<span><span class="co">#&gt; 1 0.09864399 0.09651684 0.04835076 0.1610664          NA         NA 0.95    ETI &lt;NA&gt; &lt;NA&gt; AME x</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
+<span><span class="co">#&gt;           M        Mdn         LL        UL PercentROPE PercentMID   CI CIType ROPE  MID Label</span></span>
+<span><span class="co">#&gt; 1 0.0987172 0.09704707 0.04695867 0.1609608          NA         NA 0.95    ETI &lt;NA&gt; &lt;NA&gt; AME x</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
 </div>
 <p>See the <a href="alternative_software.html">alternative software vignette</a> for more information on <code>brmsmargins</code>.</p>
 </section></section><section id="multinomial-logit" class="level2" data-number="8.3"><h2 data-number="8.3" class="anchored" data-anchor-id="multinomial-logit">

diff --git a/docs/articles/brms_files/figure-html/unnamed-chunk-16-1.png b/docs/articles/brms_files/figure-html/unnamed-chunk-16-1.png
diff --git a/docs/articles/brms_files/figure-html/unnamed-chunk-21-1.png b/docs/articles/brms_files/figure-html/unnamed-chunk-21-1.png
diff --git a/docs/articles/brms_files/figure-html/unnamed-chunk-23-1.png b/docs/articles/brms_files/figure-html/unnamed-chunk-23-1.png
diff --git a/docs/articles/brms_files/figure-html/unnamed-chunk-27-1.png b/docs/articles/brms_files/figure-html/unnamed-chunk-27-1.png
diff --git a/docs/articles/brms_files/figure-html/unnamed-chunk-28-1.png b/docs/articles/brms_files/figure-html/unnamed-chunk-28-1.png
diff --git a/docs/articles/brms_files/figure-html/unnamed-chunk-30-1.png b/docs/articles/brms_files/figure-html/unnamed-chunk-30-1.png
diff --git a/docs/articles/brms_files/figure-html/unnamed-chunk-32-1.png b/docs/articles/brms_files/figure-html/unnamed-chunk-32-1.png
diff --git a/docs/articles/brms_files/figure-html/unnamed-chunk-33-1.png b/docs/articles/brms_files/figure-html/unnamed-chunk-33-1.png
diff --git a/docs/articles/brms_files/figure-html/unnamed-chunk-34-1.png b/docs/articles/brms_files/figure-html/unnamed-chunk-34-1.png
diff --git a/docs/articles/brms_files/figure-html/unnamed-chunk-35-1.png b/docs/articles/brms_files/figure-html/unnamed-chunk-35-1.png
diff --git a/docs/articles/brms_files/figure-html/unnamed-chunk-66-1.png b/docs/articles/brms_files/figure-html/unnamed-chunk-66-1.png
diff --git a/docs/articles/categorical.html b/docs/articles/categorical.html
@@ -307,14 +307,14 @@
 </li>
           <li class="sidebar-item">
   <div class="sidebar-item-container"> 
-  <a href="../articles/machine_learning.html" class="sidebar-item-text sidebar-link">
- <span class="menu-text"><span class="chapter-number">20</span>&nbsp; <span class="chapter-title">Machine Learning</span></span></a>
+  <a href="../articles/matching.html" class="sidebar-item-text sidebar-link">
+ <span class="menu-text"><span class="chapter-number">20</span>&nbsp; <span class="chapter-title">Matching</span></span></a>
   </div>
 </li>
           <li class="sidebar-item">
   <div class="sidebar-item-container"> 
-  <a href="../articles/matching.html" class="sidebar-item-text sidebar-link">
- <span class="menu-text"><span class="chapter-number">21</span>&nbsp; <span class="chapter-title">Matching</span></span></a>
+  <a href="../articles/machine_learning.html" class="sidebar-item-text sidebar-link">
+ <span class="menu-text"><span class="chapter-number">21</span>&nbsp; <span class="chapter-title">Machine Learning</span></span></a>
   </div>
 </li>
           <li class="sidebar-item">
@@ -546,9 +546,9 @@ <h1 class="title">
 <div class="sourceCode" id="cb4"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu"><a href="https://rdrr.io/pkg/marginaleffects/man/slopes.html">slopes</a></span><span class="op">(</span><span class="va">mod</span>, variables <span class="op">=</span> <span class="st">"mpg"</span>, newdata <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/pkg/marginaleffects/man/datagrid.html">datagrid</a></span><span class="op">(</span>mpg <span class="op">=</span> <span class="fl">25</span>, hp <span class="op">=</span> <span class="fl">110</span><span class="op">)</span><span class="op">)</span></span>
 <span><span class="co">#&gt; </span></span>
 <span><span class="co">#&gt;  Group Term mpg  hp Estimate Std. Error       z Pr(&gt;|z|)    S    2.5 %  97.5 %</span></span>
-<span><span class="co">#&gt;      3  mpg  25 110 -0.06041     0.0169 -3.5813   &lt;0.001 11.5 -0.09347 -0.0273</span></span>
+<span><span class="co">#&gt;      3  mpg  25 110 -0.06041     0.0169 -3.5802   &lt;0.001 11.5 -0.09348 -0.0273</span></span>
 <span><span class="co">#&gt;      4  mpg  25 110 -0.00321     0.0335 -0.0958   0.9237  0.1 -0.06896  0.0625</span></span>
-<span><span class="co">#&gt;      5  mpg  25 110  0.06362     0.0301  2.1132   0.0346  4.9  0.00461  0.1226</span></span>
+<span><span class="co">#&gt;      5  mpg  25 110  0.06362     0.0301  2.1129   0.0346  4.9  0.00461  0.1226</span></span>
 <span><span class="co">#&gt; </span></span>
 <span><span class="co">#&gt; Columns: rowid, term, group, estimate, std.error, statistic, p.value, s.value, conf.low, conf.high, mpg, hp, predicted_lo, predicted_hi, predicted, gear </span></span>
 <span><span class="co">#&gt; Type:  probs</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
@@ -560,12 +560,12 @@ <h1 class="title">
 <div class="sourceCode" id="cb5"><pre class="downlit sourceCode r code-with-copy"><code class="sourceCode R"><span><span class="fu"><a href="https://rdrr.io/pkg/marginaleffects/man/slopes.html">avg_slopes</a></span><span class="op">(</span><span class="va">mod</span><span class="op">)</span></span>
 <span><span class="co">#&gt; </span></span>
 <span><span class="co">#&gt;  Group Term Estimate Std. Error     z Pr(&gt;|z|)    S     2.5 %   97.5 %</span></span>
-<span><span class="co">#&gt;      3  hp  -0.00377   0.001514 -2.49  0.01284  6.3 -0.006735 -0.00080</span></span>
-<span><span class="co">#&gt;      3  mpg -0.07014   0.015485 -4.53  &lt; 0.001 17.4 -0.100490 -0.03979</span></span>
-<span><span class="co">#&gt;      4  hp   0.00201   0.000957  2.10  0.03553  4.8  0.000136  0.00389</span></span>
+<span><span class="co">#&gt;      3  hp  -0.00377   0.001514 -2.49  0.01285  6.3 -0.006735 -0.00080</span></span>
+<span><span class="co">#&gt;      3  mpg -0.07014   0.015484 -4.53  &lt; 0.001 17.4 -0.100488 -0.03979</span></span>
+<span><span class="co">#&gt;      4  hp   0.00201   0.000958  2.10  0.03555  4.8  0.000136  0.00389</span></span>
 <span><span class="co">#&gt;      4  mpg  0.03747   0.013861  2.70  0.00687  7.2  0.010303  0.06464</span></span>
 <span><span class="co">#&gt;      5  hp   0.00175   0.000833  2.11  0.03519  4.8  0.000122  0.00339</span></span>
-<span><span class="co">#&gt;      5  mpg  0.03267   0.009572  3.41  &lt; 0.001 10.6  0.013907  0.05143</span></span>
+<span><span class="co">#&gt;      5  mpg  0.03267   0.009571  3.41  &lt; 0.001 10.6  0.013909  0.05143</span></span>
 <span><span class="co">#&gt; </span></span>
 <span><span class="co">#&gt; Columns: term, group, estimate, std.error, statistic, p.value, s.value, conf.low, conf.high </span></span>
 <span><span class="co">#&gt; Type:  probs</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>

diff --git a/docs/articles/comparisons.html b/docs/articles/comparisons.html
@@ -307,14 +307,14 @@
 </li>
           <li class="sidebar-item">
   <div class="sidebar-item-container"> 
-  <a href="../articles/machine_learning.html" class="sidebar-item-text sidebar-link">
- <span class="menu-text"><span class="chapter-number">20</span>&nbsp; <span class="chapter-title">Machine Learning</span></span></a>
+  <a href="../articles/matching.html" class="sidebar-item-text sidebar-link">
+ <span class="menu-text"><span class="chapter-number">20</span>&nbsp; <span class="chapter-title">Matching</span></span></a>
   </div>
 </li>
           <li class="sidebar-item">
   <div class="sidebar-item-container"> 
-  <a href="../articles/matching.html" class="sidebar-item-text sidebar-link">
- <span class="menu-text"><span class="chapter-number">21</span>&nbsp; <span class="chapter-title">Matching</span></span></a>
+  <a href="../articles/machine_learning.html" class="sidebar-item-text sidebar-link">
+ <span class="menu-text"><span class="chapter-number">21</span>&nbsp; <span class="chapter-title">Machine Learning</span></span></a>
   </div>
 </li>
           <li class="sidebar-item">

diff --git a/docs/articles/comparisons_files/figure-html/unnamed-chunk-48-1.png b/docs/articles/comparisons_files/figure-html/unnamed-chunk-48-1.png
diff --git a/docs/articles/conformal.html b/docs/articles/conformal.html
@@ -326,14 +326,14 @@
 </li>
           <li class="sidebar-item">
   <div class="sidebar-item-container"> 
-  <a href="../articles/machine_learning.html" class="sidebar-item-text sidebar-link">
- <span class="menu-text"><span class="chapter-number">20</span>&nbsp; <span class="chapter-title">Machine Learning</span></span></a>
+  <a href="../articles/matching.html" class="sidebar-item-text sidebar-link">
+ <span class="menu-text"><span class="chapter-number">20</span>&nbsp; <span class="chapter-title">Matching</span></span></a>
   </div>
 </li>
           <li class="sidebar-item">
   <div class="sidebar-item-container"> 
-  <a href="../articles/matching.html" class="sidebar-item-text sidebar-link">
- <span class="menu-text"><span class="chapter-number">21</span>&nbsp; <span class="chapter-title">Matching</span></span></a>
+  <a href="../articles/machine_learning.html" class="sidebar-item-text sidebar-link">
+ <span class="menu-text"><span class="chapter-number">21</span>&nbsp; <span class="chapter-title">Machine Learning</span></span></a>
   </div>
 </li>
           <li class="sidebar-item">

diff --git a/docs/articles/elasticity.html b/docs/articles/elasticity.html
@@ -307,14 +307,14 @@
 </li>
           <li class="sidebar-item">
   <div class="sidebar-item-container"> 
-  <a href="../articles/machine_learning.html" class="sidebar-item-text sidebar-link">
- <span class="menu-text"><span class="chapter-number">20</span>&nbsp; <span class="chapter-title">Machine Learning</span></span></a>
+  <a href="../articles/matching.html" class="sidebar-item-text sidebar-link">
+ <span class="menu-text"><span class="chapter-number">20</span>&nbsp; <span class="chapter-title">Matching</span></span></a>
   </div>
 </li>
           <li class="sidebar-item">
   <div class="sidebar-item-container"> 
-  <a href="../articles/matching.html" class="sidebar-item-text sidebar-link">
- <span class="menu-text"><span class="chapter-number">21</span>&nbsp; <span class="chapter-title">Matching</span></span></a>
+  <a href="../articles/machine_learning.html" class="sidebar-item-text sidebar-link">
+ <span class="menu-text"><span class="chapter-number">21</span>&nbsp; <span class="chapter-title">Machine Learning</span></span></a>
   </div>
 </li>
           <li class="sidebar-item">
@@ -536,7 +536,7 @@ <h1 class="title">
 <span><span class="co">#&gt; </span></span>
 <span><span class="co">#&gt;  Term Estimate Std. Error     z Pr(&gt;|z|)    S   2.5 %  97.5 %</span></span>
 <span><span class="co">#&gt;    hp  -0.0318    0.00903 -3.52   &lt;0.001 11.2 -0.0495 -0.0141</span></span>
-<span><span class="co">#&gt;    wt  -3.8778    0.63276 -6.13   &lt;0.001 30.1 -5.1180 -2.6376</span></span>
+<span><span class="co">#&gt;    wt  -3.8778    0.63273 -6.13   &lt;0.001 30.1 -5.1180 -2.6377</span></span>
 <span><span class="co">#&gt; </span></span>
 <span><span class="co">#&gt; Columns: term, estimate, std.error, statistic, p.value, s.value, conf.low, conf.high </span></span>
 <span><span class="co">#&gt; Type:  response</span></span>
@@ -554,7 +554,7 @@ <h1 class="title">
 <span><span class="co">#&gt; </span></span>
 <span><span class="co">#&gt;  Term Contrast Estimate Std. Error     z Pr(&gt;|z|)    S    2.5 %    97.5 %</span></span>
 <span><span class="co">#&gt;    hp    eY/dX -0.00173   0.000502 -3.46   &lt;0.001 10.8 -0.00272 -0.000751</span></span>
-<span><span class="co">#&gt;    wt    eY/dX -0.21165   0.037851 -5.59   &lt;0.001 25.4 -0.28583 -0.137461</span></span>
+<span><span class="co">#&gt;    wt    eY/dX -0.21165   0.037849 -5.59   &lt;0.001 25.4 -0.28583 -0.137464</span></span>
 <span><span class="co">#&gt; </span></span>
 <span><span class="co">#&gt; Columns: term, contrast, estimate, std.error, statistic, p.value, s.value, conf.low, conf.high </span></span>
 <span><span class="co">#&gt; Type:  response</span></span>

diff --git a/docs/articles/equivalence.html b/docs/articles/equivalence.html
@@ -307,14 +307,14 @@
 </li>
           <li class="sidebar-item">
   <div class="sidebar-item-container"> 
-  <a href="../articles/machine_learning.html" class="sidebar-item-text sidebar-link">
- <span class="menu-text"><span class="chapter-number">20</span>&nbsp; <span class="chapter-title">Machine Learning</span></span></a>
+  <a href="../articles/matching.html" class="sidebar-item-text sidebar-link">
+ <span class="menu-text"><span class="chapter-number">20</span>&nbsp; <span class="chapter-title">Matching</span></span></a>
   </div>
 </li>
           <li class="sidebar-item">
   <div class="sidebar-item-container"> 
-  <a href="../articles/matching.html" class="sidebar-item-text sidebar-link">
- <span class="menu-text"><span class="chapter-number">21</span>&nbsp; <span class="chapter-title">Matching</span></span></a>
+  <a href="../articles/machine_learning.html" class="sidebar-item-text sidebar-link">
+ <span class="menu-text"><span class="chapter-number">21</span>&nbsp; <span class="chapter-title">Machine Learning</span></span></a>
   </div>
 </li>
           <li class="sidebar-item">