diff --git a/README.md b/README.md index e31dd8a..7cba439 100644 --- a/README.md +++ b/README.md @@ -40,8 +40,8 @@ In your markdown files you can now use: Where the path is relative to the location of your project's `mkdocs.yml` file, _or_ your project's `docs/` directory, _or_ the location of your markdown source file (all 3 possible locations will be searched, in that order). -- There are [readers](https://timvink.github.io/mkdocs-table-reader-plugin/readers/) for `.csv`, `.fwf`, `.json`, `.xls`, `.xlsx`, `.yaml`, `.feather` and `.tsv` files. There is also the `read_raw()` reader that will allow you to insert tables (or other content) already in markdown format. -- `table-reader` is compatible with [`mkdocs-macros-plugin`](https://mkdocs-macros-plugin.readthedocs.io/en/latest/), which means you can [dynamically insert tables using jinja2 syntax](https://timvink.github.io/mkdocs-table-reader-plugin/howto/use_jinja2/). +- There are [readers](https://timvink.github.io/mkdocs-table-reader-plugin/readers/) available for many common table formats, like `.csv`, `.fwf`, `.json`, `.xls`, `.xlsx`, `.yaml`, `.feather` and `.tsv`. There is also the `read_raw()` reader that will allow you to insert tables (or other content) already in markdown format. +- `table-reader` is compatible with [`mkdocs-macros-plugin`](https://mkdocs-macros-plugin.readthedocs.io/en/latest/). This enables further automation like filtering tables or inserting directories of tables. See the documentation on [compatibility with macros plugin](howto/use_jinja2.md) for more examples. ## Documentation and how-to guides diff --git a/docs/howto/alternatives.md b/docs/howto/alternatives.md index 4f7ad33..cec6bbf 100644 --- a/docs/howto/alternatives.md +++ b/docs/howto/alternatives.md @@ -39,7 +39,7 @@ Downsides: ## Execute python during build -You could also choose to insert the markdown for tables dynamically, using packages like [markdown-exec]() or [mkdocs-macros-plugin](https://mkdocs-macros-plugin.readthedocs.io/). +You could also choose to insert the markdown for tables dynamically, using packages like [markdown-exec](https://pypi.org/project/markdown-exec/). For example: diff --git a/docs/howto/customize_tables.md b/docs/howto/customize_tables.md index 02dac1c..66c1312 100644 --- a/docs/howto/customize_tables.md +++ b/docs/howto/customize_tables.md @@ -12,8 +12,10 @@ df = pd.read_csv('path_to_table.csv') df.to_markdown(index=False, tablefmt='pipe') ``` -Any keyword arguments you give to \{\{ read_csv('path_to_your_table.csv') \}\} will be matched and passed the corresponding [pandas.read_csv()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html) and/or +{% raw %} +Any keyword arguments you give to `{{ read_csv('path_to_your_table.csv') }}` will be matched and passed the corresponding [pandas.read_csv()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html) and/or [.to_markdown()](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_markdown.html) functions. +{% endraw %} Pandas's `.to_markdown()` uses the [tabulate](https://pypi.org/project/tabulate/) package and any keyword arguments that are passed to it. Tabulate in turn offers many customization options, see [library usage](https://github.com/astanin/python-tabulate#library-usage). @@ -23,21 +25,33 @@ Text columns will be aligned to the left [by default](https://github.com/astanin === ":arrow_left: left" - \{\{ read_csv('tables/basic_table.csv', colalign=("left",)) \}\} - + {% raw %} + ``` {{ read_csv('tables/basic_table.csv', colalign=("left",)) }} + ``` + {% endraw %} -=== ":left_right_arrow: center" + {{ read_csv('tables/basic_table.csv', colalign=("left",)) | add_indentation(spaces=4) }} - \{\{ read_csv('tables/basic_table.csv', colalign=("center",)) \}\} +=== ":left_right_arrow: center" + {% raw %} + ``` {{ read_csv('tables/basic_table.csv', colalign=("center",)) }} + ``` + {% endraw %} -=== ":arrow_right: right" + {{ read_csv('tables/basic_table.csv', colalign=("center",)) | add_indentation(spaces=4) }} - \{\{ read_csv('tables/basic_table.csv', colalign=("right",)) \}\} +=== ":arrow_right: right" + {% raw %} + ``` {{ read_csv('tables/basic_table.csv', colalign=("right",)) }} + ``` + {% endraw %} + + {{ read_csv('tables/basic_table.csv', colalign=("right",)) | add_indentation(spaces=4) }} ## Sortable tables @@ -47,21 +61,33 @@ If you use [mkdocs-material](https://squidfunk.github.io/mkdocs-material), you c You can use [tabulate](https://pypi.org/project/tabulate/)'s [number formatting](https://github.com/astanin/python-tabulate#number-formatting). Example: -=== ":zero:" - - \{\{ read_fwf('tables/fixedwidth_table.txt', floatfmt=".0f") \}\} +=== "zero points" + {% raw %} + ``` {{ read_fwf('tables/fixedwidth_table.txt', floatfmt=".0f") }} + ``` + {% endraw %} -=== ":one:" + {{ read_fwf('tables/fixedwidth_table.txt', floatfmt=".0f") | add_indentation(spaces=4) }} - \{\{ read_fwf('tables/fixedwidth_table.txt', floatfmt=".1f") \}\} +=== "one points" + {% raw %} + ``` {{ read_fwf('tables/fixedwidth_table.txt', floatfmt=".1f") }} + ``` + {% endraw %} -=== ":two:" + {{ read_fwf('tables/fixedwidth_table.txt', floatfmt=".1f") | add_indentation(spaces=4) }} - \{\{ read_fwf('tables/fixedwidth_table.txt', floatfmt=".2f") \}\} +=== "two points" + {% raw %} + ``` {{ read_fwf('tables/fixedwidth_table.txt', floatfmt=".2f") }} + ``` + {% endraw %} + + {{ read_fwf('tables/fixedwidth_table.txt', floatfmt=".2f") | add_indentation(spaces=4) }} diff --git a/docs/howto/preprocess_tables.md b/docs/howto/preprocess_tables.md index e1c2558..a150e35 100644 --- a/docs/howto/preprocess_tables.md +++ b/docs/howto/preprocess_tables.md @@ -1,5 +1,7 @@ # Preprocess input tables +{% raw %} + `mkdocs>=1.4` supports [hooks](https://www.mkdocs.org/user-guide/configuration/#hooks), which enable you to run python scripts on `mkdocs serve` or `mkdocs build`. Here are some example of workflows that use hooks and the `table-reader` plugin: @@ -26,7 +28,7 @@ Here are some example of workflows that use hooks and the `table-reader` plugin: My table: - \{\{ read_csv("docs/assets/output_table.csv") \}\} + {{ read_csv("docs/assets/output_table.csv") }} === "mkdocs.yml" @@ -74,7 +76,7 @@ Here are some example of workflows that use hooks and the `table-reader` plugin: My table: - \{\{ read_csv("docs/assets/nyc_data.csv") \}\} + {{ read_csv("docs/assets/nyc_data.csv") }} === "mkdocs.yml" @@ -101,3 +103,5 @@ Here are some example of workflows that use hooks and the `table-reader` plugin: ``` Note that during development when you use `mkdocs serve` and autoreload, you might not want to run this hook every time you make a change. You could use an environment variable inside your hook, for example something like `if os.environ['disable_hook'] == 1: return None`. + +{% endraw %} \ No newline at end of file diff --git a/docs/howto/project_structure.md b/docs/howto/project_structure.md index 7ba2966..9243a18 100644 --- a/docs/howto/project_structure.md +++ b/docs/howto/project_structure.md @@ -1,4 +1,5 @@ # Choose a project structure +{% raw %} You have different possible strategies to store and load your tables. This guide gives some examples. @@ -23,14 +24,14 @@ If you only want to include an occasional table in a specific markdown file, jus ```md Here is the table: - \{\{ read_csv("another_table.csv") \}\} + {{ read_csv("another_table.csv") }} ``` In `page.md`, to read `another_table.csv`, you can choose to use: -- \{\{ read_csv("docs/folder/another_table.csv") \}\} (Path relative to mkdocs.yml) -- \{\{ read_csv("folder/another_table.csv") \}\} (Path relative to docs/ directory) -- \{\{ read_csv("another_table.csv") \}\} (Path relative to page source file) +- `{{ read_csv("docs/folder/another_table.csv") }}` (Path relative to mkdocs.yml) +- `{{ read_csv("folder/another_table.csv") }}` (Path relative to docs/ directory) +- `{{ read_csv("another_table.csv") }}` (Path relative to page source file) ## Re-using tables across markdown files @@ -54,7 +55,8 @@ Given the following project structure: In `page.md`, to read `another_table.csv`, you can choose to use: -- \{\{ read_csv("docs/assets/tables/another_table.csv") \}\} (Path relative to mkdocs.yml) -- \{\{ read_csv("assets/tables/another_table.csv") \}\} (Path relative to docs/ directory) -- \{\{ read_csv("../assets/tables/another_table.csv") \}\} (Path relative to page source file _(note that `..` stands for "one directory up")_) +- `{{ read_csv("docs/assets/tables/another_table.csv") }}` (Path relative to mkdocs.yml) +- `{{ read_csv("assets/tables/another_table.csv") }}` (Path relative to docs/ directory) +- `{{ read_csv("../assets/tables/another_table.csv") }}` (Path relative to page source file _(note that `..` stands for "one directory up")_) +{% endraw %} \ No newline at end of file diff --git a/docs/howto/use_jinja2.md b/docs/howto/use_jinja2.md index 8ab66f9..f1adc10 100644 --- a/docs/howto/use_jinja2.md +++ b/docs/howto/use_jinja2.md @@ -1,4 +1,6 @@ -# Use jinja2 for automation +# Compatibility with mkdocs-macros-plugin to enable further automation + +{% raw %} `table-reader` supports [`mkdocs-macros-plugin`](https://mkdocs-macros-plugin.readthedocs.io/en/latest/), which enables you to use jinja2 syntax inside markdown files (among other things). @@ -10,26 +12,19 @@ plugins: - table-reader ``` -Now you can do cool things like: - -## Dynamically load a list of tables +Everything will work as before, _except_ indentation will not be retrained. This means components that rely on indentation (like [Admonitions](https://squidfunk.github.io/mkdocs-material/reference/admonitions/) and [Content tabs](https://squidfunk.github.io/mkdocs-material/reference/content-tabs/#usage)) will break. -```markdown -# index.md +The solution is to use the custom _filter_ `add_indendation` (a filter added to `macros` by `table-reader` plugin, see the [readers](../readers.md)). For example: -{% set table_names = ["basic_table.csv","basic_table2.csv"] %} -{% for table_name in table_names %} +```jinja +!!! note "This is a note" -{ { read_csv(table_name) }} - -{% endfor %} + {{ read_csv("basic_table.csv") | add_indentation(spaces=4) }} ``` -## Insert tables into content tabs - -If you inserted content has multiple lines, then indentation will be not be retained beyond the first line. This means things like [content tabs](https://squidfunk.github.io/mkdocs-material/reference/content-tabs/#usage) will not work as expected. +The upside is you now have much more control. A couple of example use cases: -To fix that, you can use the custom _filter_ `add_indendation` (a filter add to `macros` by `table-reader` plugin). For example: +## Dynamically load a specified list of tables into tabs === "index.md" @@ -39,7 +34,7 @@ To fix that, you can use the custom _filter_ `add_indendation` (a filter add to === "{{ table_name }}" - { { read_csv(table_name) | add_indentation(spaces=4) }} + {{ read_csv(table_name) | add_indentation(spaces=4) }} {% endfor %} ``` @@ -64,11 +59,6 @@ To fix that, you can use the custom _filter_ `add_indendation` (a filter add to alternate_style: true ``` -!!! note "Note the space in { {" - - To avoid the tables being inserted into the code example, we replaced `{{` with `{ {`. - If you copy this example, make sure to fix. - ## Recursively insert an entire directory of tables @@ -100,12 +90,19 @@ Now you could do something like: {% for table_name in listdir('docs/assets/my_tables") %} -{ { read_csv(table_name) }} +{{ read_csv(table_name) }} {% endfor %} ``` -!!! note "Note the space in { {" +## Filter a table before inserting it + +When you enable the `macros` plugin in your `mkdocs.yml`, `table-reader` will add additional _macros_ and _filters_ (see the [readers overview](../readers.md)). + +You can use the `pd_` variants to read a file to a pandas dataframe. Then you can use the pandas methods like [`.query()`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.query.html). For example: + +``` +{{ pd_read_csv("numeric_table.csv").query("column_name >= 3") | convert_to_md_table }} +``` - To avoid the tables being inserted into the code example, we replaced `{{` with `{ {`. - If you copy this example, make sure to fix. \ No newline at end of file +{% endraw %} diff --git a/docs/options.md b/docs/options.md index 0853e17..52e19a5 100644 --- a/docs/options.md +++ b/docs/options.md @@ -4,6 +4,7 @@ hide: --- # Options +{% raw %} You can customize the plugin by setting options in `mkdocs.yml`. For example: @@ -24,7 +25,7 @@ Default is `.`. Set a default path to the searched directories in order to short Given a file path, `table-reader` will search for that file relative to your your project's `mkdocs.yml` and relative to your `docs/` folder. If you use a folder for all your table files you can shorten the path specification by setting the `data_path`. -For example, if your table is located in `docs/assets/tables/basic_table.csv`, you can set `data_path` to `docs/assets/tables/`. Then you will be able to use \{\{ read_csv("basic_table.csv") \}\} instead of \{\{ read_csv("docs/assets/tables/basic_table.csv") \}\} inside any markdown page. +For example, if your table is located in `docs/assets/tables/basic_table.csv`, you can set `data_path` to `docs/assets/tables/`. Then you will be able to use `{{ read_csv("basic_table.csv") }}` instead of `{{ read_csv("docs/assets/tables/basic_table.csv") }}` inside any markdown page. !!! info @@ -38,7 +39,7 @@ Default: `False`. When enabled, if a filepath is not found, the plugin will rais ## `select_readers` -Default: Selects all available readers. Specify a list of readers to improve documentation build times for very large sites. +Default: Selects all available readers. Specify a list of readers to improve documentation build times for very large sites. This option is ignored when you use this plugin with `mkdocs-macros-plugin` ([read more](howto/use_jinja2.md)) ## `enabled` @@ -57,4 +58,6 @@ Which enables you to disable the plugin locally using: ```bash export ENABLED_TABLE_READER=false mkdocs serve -``` \ No newline at end of file +``` + +{% endraw %} \ No newline at end of file diff --git a/docs/readers.md b/docs/readers.md index 5298fd6..50ed597 100644 --- a/docs/readers.md +++ b/docs/readers.md @@ -5,116 +5,394 @@ hide: # Readers +## Basic readers + The following table reader functions are available: -## read_csv +### `read_csv` + +Use {% raw %}`{{ read_csv() }}`{% endraw %} to read a comma-separated values (csv) and output as a markdown table. -`{{ read_csv() }}` passed to [pandas.read_csv()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html). Example: +1. Arguments are parsed safely and then passed to corresponding functions below +2. File is read using [pandas.read_csv()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html) +3. The `pd.DataFrame` is then converted to a markdown table using [`.to_markdown()`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_markdown.html) +4. The markdown table is fixed to match the indentation used by the tag in the markdown document (only when _not_ used with `mkdocs-macros-plugin`. See [compatibility with macros plugin](howto/use_jinja2.md)) + +Example: === "Input" - \{\{ read_csv('tables/basic_table.csv') \}\} + {% raw %} + ```markdown + {{ read_csv('assets/tables/basic_table.csv') }} + ``` + {% endraw %} === "Output" - {{ read_csv('tables/basic_table.csv') }} + {{ read_csv('assets/tables/basic_table.csv') | add_indentation(spaces=4) }} -## read_fwf +### `read_fwf` -`{{ read_fwf() }}` passed to [pandas.read_fwf()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_fwf.html). Example: +Use {% raw %}`{{ read_fwf() }}`{% endraw %} to read a table of fixed-width formatted lines and output as a markdown table. + +1. Arguments are parsed safely and then passed to corresponding functions below +2. File is read using [pandas.read_fwf()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_fwf.html) +3. The `pd.DataFrame` is then converted to a markdown table using [`.to_markdown()`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_markdown.html) +4. The markdown table is fixed to match the indentation used by the tag in the markdown document (only when _not_ used with `mkdocs-macros-plugin`. See [compatibility with macros plugin](howto/use_jinja2.md)) + +Example: === "Input" - \{\{ read_fwf('tables/fixedwidth_table.txt') \}\} + {% raw %} + ```markdown + {{ read_fwf('assets/tables/fixedwidth_table.txt') }} + ``` + {% endraw %} === "Output" - {{ read_fwf('tables/fixedwidth_table.txt') }} + {{ read_fwf('assets/tables/fixedwidth_table.txt') | add_indentation(spaces=4) }} + +### `read_yaml` -## read_yaml +Use {% raw %}`{{ read_yaml() }}`{% endraw %} to read a YAML file and output as a markdown table. -`{{ read_yaml() }}` is parsed with [yaml.safe_load()](https://pyyaml.org/wiki/PyYAMLDocumentation#loading-yaml) and passed to [pandas.json_normalize()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.json_normalize.html). Example: +1. Arguments are parsed safely and then passed to corresponding functions below +2. File is read using [yaml.safe_load()](https://pyyaml.org/wiki/PyYAMLDocumentation#loading-yaml) and then passed to [pandas.json_normalize()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.json_normalize.html) +3. The `pd.DataFrame` is then converted to a markdown table using [`.to_markdown()`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_markdown.html) +4. The markdown table is fixed to match the indentation used by the tag in the markdown document (only when _not_ used with `mkdocs-macros-plugin`. See [compatibility with macros plugin](howto/use_jinja2.md)) + +Example: === "Input" - \{\{ read_yaml('tables/yaml_table.yml') \}\} + {% raw %} + ```markdown + {{ read_yaml('assets/tables/yaml_table.yml') }} + ``` + {% endraw %} === "Output" - {{ read_yaml('tables/yaml_table.yml') }} + {{ read_yaml('assets/tables/yaml_table.yml') | add_indentation(spaces=4) }} + + +### `read_table` +Use {% raw %}`{{ read_table() }}`{% endraw %} to read a general delimited file and output as a markdown table. -## read_table +1. Arguments are parsed safely and then passed to corresponding functions below +2. File is read using [pandas.read_table()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_table.html) +3. The `pd.DataFrame` is then converted to a markdown table using [`.to_markdown()`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_markdown.html) +4. The markdown table is fixed to match the indentation used by the tag in the markdown document (only when _not_ used with `mkdocs-macros-plugin`. See [compatibility with macros plugin](howto/use_jinja2.md)) -`{{ read_table() }}` passed to [pandas.read_table()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_table.html). Example: +Example: === "Input" - \{\{ read_table('tables/basic_table.csv', sep = ',') \}\} + {% raw %} + ```markdown + {{ read_table('assets/tables/basic_table.csv', sep = ',') }} + ``` + {% endraw %} === "Output" - {{ read_table('tables/basic_table.csv', sep = ',') }} + {{ read_table('assets/tables/basic_table.csv', sep = ',') | add_indentation(spaces=4) }} + +### `read_json` -## read_json +Use {% raw %}`{{ read_json() }}`{% endraw %} to read a JSON string path and output as a markdown table. -`{{ read_json() }}` passed to [pandas.read_json()](https://pandas.pydata.org/docs/reference/api/pandas.read_json.html). Example: +1. Arguments are parsed safely and then passed to corresponding functions below +2. File is read using [pandas.read_json()](https://pandas.pydata.org/docs/reference/api/pandas.read_json.html) +3. The `pd.DataFrame` is then converted to a markdown table using [`.to_markdown()`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_markdown.html) +4. The markdown table is fixed to match the indentation used by the tag in the markdown document (only when _not_ used with `mkdocs-macros-plugin`. See [compatibility with macros plugin](howto/use_jinja2.md)) + +Example: === "Input" - \{\{ read_json('tables/data.json', orient='split') \}\} + {% raw %} + ```markdown + {{ read_json('assets/tables/data.json', orient='split') }} + ``` + {% endraw %} === "Output" - {{ read_json('tables/data.json', orient='split') }} + {{ read_json('assets/tables/data.json', orient='split') | add_indentation(spaces=4) }} -## read_feather +### `read_feather` -`{{ read_feather() }}` passed to [pandas.read_feather()](https://pandas.pydata.org/docs/reference/api/pandas.read_feather.html). Example: +Use {% raw %}`{{ read_feather() }}`{% endraw %} to read a feather-format object and output as a markdown table. + +1. Arguments are parsed safely and then passed to corresponding functions below +2. File is read using [pandas.read_feather()](https://pandas.pydata.org/docs/reference/api/pandas.read_feather.html) +3. The `pd.DataFrame` is then converted to a markdown table using [`.to_markdown()`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_markdown.html) +4. The markdown table is fixed to match the indentation used by the tag in the markdown document (only when _not_ used with `mkdocs-macros-plugin`. See [compatibility with macros plugin](howto/use_jinja2.md)) + +Example: === "Input" - \{\{ read_json('tables/data.feather') \}\} + {% raw %} + ```markdown + {{ read_feather('assets/tables/data.feather') }} + ``` + {% endraw %} === "Output" - {{ read_feather('tables/data.feather') }} + {{ read_feather('assets/tables/data.feather') | add_indentation(spaces=4) }} + +### `read_excel` -## read_excel +Use {% raw %}`{{ read_excel() }}`{% endraw %} to read an Excel file and output as a markdown table. -`{{ read_excel() }}` passed to [pandas.read_excel()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_excel.html). Example: +1. Arguments are parsed safely and then passed to corresponding functions below +2. File is read using [pandas.read_excel()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_excel.html) +3. The `pd.DataFrame` is then converted to a markdown table using [`.to_markdown()`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_markdown.html) +4. The markdown table is fixed to match the indentation used by the tag in the markdown document (only when _not_ used with `mkdocs-macros-plugin`. See [compatibility with macros plugin](howto/use_jinja2.md)) +Example: === "Input" - \{\{ read_excel('tables/excel_table.xlsx', engine='openpyxl') \}\} + {% raw %} + ```markdown + {{ read_excel('assets/tables/excel_table.xlsx', engine='openpyxl') }} + ``` + {% endraw %} === "Output" - {{ read_excel('tables/excel_table.xlsx', engine='openpyxl') }} + {{ read_excel('assets/tables/excel_table.xlsx', engine='openpyxl') | add_indentation(spaces=4) }} !!! info "Reading xlsx files" You might get a `XLRDError('Excel xlsx file; not supported',)` error when trying to read modern excel files. That's because `xlrd` does not support `.xlsx` files ([stackoverflow post](https://stackoverflow.com/questions/65254535/xlrd-biffh-xlrderror-excel-xlsx-file-not-supported)). Instead, install [openpyxl](https://openpyxl.readthedocs.io/en/stable/) and use: - \{\{ read_excel('tables/excel_table.xlsx', engine='openpyxl') \}\} + {% raw %} + ```markdown + {{ read_excel('assets/tables/excel_table.xlsx', engine='openpyxl') }} + ``` + {% endraw %} + +### `read_raw` -## read_raw +Use {% raw %}`{{ read_raw() }}`{% endraw %} to insert the contents from a file directly. -`{{ read_raw() }}` inserts contents from a file directly. This is great if you have a file with a table already in markdown format. +This is great if you have a file with a table already in markdown format. It could also replace a workflow where you use the [snippets extension to embed external files](https://squidfunk.github.io/mkdocs-material/reference/code-blocks/#embedding-external-files). +1. Only the first argument is read. This should be the file path. +2. File is read using python +4. The markdown table is fixed to match the indentation used by the tag in the markdown document (only when _not_ used with `mkdocs-macros-plugin`. See [compatibility with macros plugin](howto/use_jinja2.md)) + +Example: + +=== "Input" + + {% raw %} + ```markdown + {{ read_raw('assets/tables/markdown_table.md') }} + ``` + {% endraw %} + +=== "Output" + + {{ read_raw('assets/tables/markdown_table.md') | add_indentation(spaces=4) }} + + +## Macros + +When you use `table-reader` with [`mkdocs-macros-plugin`](https://mkdocs-macros-plugin.readthedocs.io/en/latest/), in next to all the readers, the following _additional_ macros will be made available: + +### `pd_read_csv` + +Use {% raw %}`{{ pd_read_csv() }}`{% endraw %} to read a comma-separated values (csv) using [pandas.read_csv()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html). + +=== "Input" + + {% raw %} + ```markdown + {{ pd_read_csv('assets/tables/basic_table.csv').to_markdown(tablefmt="pipe", index=False) | add_indentation(spaces=4) }} + ``` + {% endraw %} + +=== "Output" + + {{ pd_read_csv('assets/tables/basic_table.csv').to_markdown(tablefmt="pipe", index=False) | add_indentation(spaces=4) }} + + +### `pd_read_fwf` + +Use {% raw %}`{{ pd_read_fwf() }}`{% endraw %} to read a table of fixed-width formatted lines using [pandas.read_fwf()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_fwf.html) + +Example: + +=== "Input" + + {% raw %} + ```markdown + {{ pd_read_fwf('assets/tables/fixedwidth_table.txt').to_markdown(tablefmt="pipe", index=False) | add_indentation(spaces=4) }} + ``` + {% endraw %} + +=== "Output" + + {{ pd_read_fwf('assets/tables/fixedwidth_table.txt').to_markdown(tablefmt="pipe", index=False) | add_indentation(spaces=4) }} + + +### `pd_read_yaml` + +Use {% raw %}`{{ pd_read_yaml() }}`{% endraw %} to read a YAML file using [yaml.safe_load()](https://pyyaml.org/wiki/PyYAMLDocumentation#loading-yaml) and [pandas.json_normalize()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.json_normalize.html). + +Example: + +=== "Input" + + {% raw %} + ```markdown + {{ pd_read_yaml('assets/tables/yaml_table.yml').to_markdown(tablefmt="pipe", index=False) | add_indentation(spaces=4) }} + ``` + {% endraw %} + +=== "Output" + + {{ pd_read_yaml('assets/tables/yaml_table.yml').to_markdown(tablefmt="pipe", index=False) | add_indentation(spaces=4) }} + + +### `pd_read_table` + +Use {% raw %}`{{ pd_read_table() }}`{% endraw %} to read a general delimited file using [pandas.read_table()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_table.html). + +Example: + +=== "Input" + + {% raw %} + ```markdown + {{ pd_read_table('assets/tables/basic_table.csv', sep = ',').to_markdown(tablefmt="pipe", index=False) | add_indentation(spaces=4) }} + ``` + {% endraw %} + +=== "Output" + + {{ pd_read_table('assets/tables/basic_table.csv', sep = ',').to_markdown(tablefmt="pipe", index=False) | add_indentation(spaces=4) }} + + + +### `pd_read_json` + +Use {% raw %}`{{ pd_read_json() }}`{% endraw %} to read a JSON string path using [pandas.read_json()](https://pandas.pydata.org/docs/reference/api/pandas.read_json.html) + +Example: + +=== "Input" + + {% raw %} + ```markdown + {{ pd_read_json('assets/tables/data.json', orient='split').to_markdown(tablefmt="pipe", index=False) | add_indentation(spaces=4) }} + ``` + {% endraw %} + +=== "Output" + + {{ pd_read_json('assets/tables/data.json', orient='split').to_markdown(tablefmt="pipe", index=False) | add_indentation(spaces=4) }} + + + +### `pd_read_feather` + +Use {% raw %}`{{ pd_read_feather() }}`{% endraw %} to read a feather-format object using [pandas.read_feather()](https://pandas.pydata.org/docs/reference/api/pandas.read_feather.html) + + Example: === "Input" - \{\{ read_raw('tables/markdown_table.md') \}\} + {% raw %} + ```markdown + {{ pd_read_feather('assets/tables/data.feather').to_markdown(tablefmt="pipe", index=False) | add_indentation(spaces=4) }} + ``` + {% endraw %} === "Output" - {{ read_raw('tables/markdown_table.md') }} + {{ pd_read_feather('assets/tables/data.feather').to_markdown(tablefmt="pipe", index=False) | add_indentation(spaces=4) }} + + +### `pd_read_excel` + +Use {% raw %}`{{ pd_read_excel() }}`{% endraw %} to read an Excel file using [pandas.read_excel()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_excel.html) + +Example: + +=== "Input" + + {% raw %} + ```markdown + {{ pd_read_excel('assets/tables/excel_table.xlsx', engine='openpyxl').to_markdown(tablefmt="pipe", index=False) | add_indentation(spaces=4) }} + ``` + {% endraw %} + +=== "Output" + + {{ pd_read_excel('assets/tables/excel_table.xlsx', engine='openpyxl').to_markdown(tablefmt="pipe", index=False) | add_indentation(spaces=4) }} + + +!!! info "Reading xlsx files" + + You might get a `XLRDError('Excel xlsx file; not supported',)` error when trying to read modern excel files. That's because `xlrd` does not support `.xlsx` files ([stackoverflow post](https://stackoverflow.com/questions/65254535/xlrd-biffh-xlrderror-excel-xlsx-file-not-supported)). Instead, install [openpyxl](https://openpyxl.readthedocs.io/en/stable/) and use: + + {% raw %} + ```markdown + {{ pd_read_excel('assets/tables/excel_table.xlsx', engine='openpyxl') }} + ``` + {% endraw %} + + +## Filters + +When you use `table-reader` with [`mkdocs-macros-plugin`](https://mkdocs-macros-plugin.readthedocs.io/en/latest/), in next to all the readers, the macros, the following _additional_ filters will be made available: + +### `add_indentation` + +Adds a consistent indentation to every line in a string. This is important when you are inserting content into [Admonitions](https://squidfunk.github.io/mkdocs-material/reference/admonitions/) or [Content tabs](https://squidfunk.github.io/mkdocs-material/reference/content-tabs/). + +Args: + text (str): input text + spaces (int): Indentation to add in spaces + tabs (int): Indentation to add in tabs + + +Example usage: + +{% raw %} +```markdown +!!! note "this is a note" + {{ pd_read_csv('assets/tables/basic_table.csv').to_markdown(tablefmt="pipe", index=False) | add_indentation(spaces=4) }} +``` +{% endraw %} + +### `convert_to_md_table` + +Converts a pandas dataframe into a markdown table. Arguments are passed to [`.to_markdown()`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_markdown.html). By default, `tablefmt='pipe'` and `index=False` are used. +There is also an additional fix to ensure any pipe (`|`) characters in the dataframe are properly escaped ([python-tabulate#241](https://github.com/astanin/python-tabulate/issues/241)). + +Example usage: + +{% raw %} +```markdown +{{ pd_read_csv('assets/tables/basic_table.csv') | convert_to_md_table }} +``` +{% endraw %} diff --git a/mkdocs.yml b/mkdocs.yml index 4240b3c..80b134c 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -10,7 +10,7 @@ nav: - How to: - howto/customize_tables.md - howto/preprocess_tables.md - - howto/use_jinja2.md + - Macros plugin compatibility: howto/use_jinja2.md - howto/project_structure.md - howto/docker.md - howto/alternatives.md @@ -48,6 +48,7 @@ theme: plugins: - search + - macros - table-reader: data_path: "docs/assets" - git-authors: diff --git a/mkdocs_table_reader_plugin/markdown.py b/mkdocs_table_reader_plugin/markdown.py index 919d09e..bc66c6f 100644 --- a/mkdocs_table_reader_plugin/markdown.py +++ b/mkdocs_table_reader_plugin/markdown.py @@ -19,7 +19,7 @@ def replace_unescaped_pipes(text: str) -> str: return re.sub(r"(? str: +def convert_to_md_table(df: pd.DataFrame, **markdown_kwargs: Dict) -> str: """ Convert dataframe to markdown table using tabulate. """ diff --git a/mkdocs_table_reader_plugin/plugin.py b/mkdocs_table_reader_plugin/plugin.py index 444d474..62ab21b 100644 --- a/mkdocs_table_reader_plugin/plugin.py +++ b/mkdocs_table_reader_plugin/plugin.py @@ -5,8 +5,8 @@ from mkdocs.exceptions import ConfigurationError from mkdocs_table_reader_plugin.safe_eval import parse_argkwarg -from mkdocs_table_reader_plugin.readers import READERS -from mkdocs_table_reader_plugin.markdown import fix_indentation, add_indentation +from mkdocs_table_reader_plugin.readers import READERS, MACROS +from mkdocs_table_reader_plugin.markdown import fix_indentation, add_indentation, convert_to_md_table logger = get_plugin_logger("table-reader") @@ -70,13 +70,23 @@ def on_config(self, config, **kwargs): ) if "macros" in config.plugins: - config.plugins["macros"].macros.update(self.readers) - config.plugins["macros"].variables["macros"].update(self.readers) - config.plugins["macros"].env.globals.update(self.readers) - - config.plugins["macros"].filters.update({"add_indentation": add_indentation}) - config.plugins["macros"].variables["filters"].update({"add_indentation": add_indentation}) - config.plugins["macros"].env.filters.update({"add_indentation": add_indentation}) + self.macros = { + macro: MACROS[macro].set_config_context( + mkdocs_config=config, plugin_config=self.config + ) + for macro in MACROS + } + self.filters = { + "add_indentation": add_indentation, + "convert_to_md_table": convert_to_md_table, + } + config.plugins["macros"].macros.update(self.macros) + config.plugins["macros"].variables["macros"].update(self.macros) + config.plugins["macros"].env.globals.update(self.macros) + + config.plugins["macros"].filters.update(self.filters) + config.plugins["macros"].variables["filters"].update(self.filters) + config.plugins["macros"].env.filters.update(self.filters) self.external_jinja_engine = True else: diff --git a/mkdocs_table_reader_plugin/readers.py b/mkdocs_table_reader_plugin/readers.py index 7794625..2144d62 100644 --- a/mkdocs_table_reader_plugin/readers.py +++ b/mkdocs_table_reader_plugin/readers.py @@ -60,22 +60,38 @@ def __call__(self, *args, **kwargs): return self.func(valid_file_paths[0], *args, **kwargs) +@ParseArgs +def pd_read_csv(*args, **kwargs) -> str: + read_kwargs = kwargs_in_func(kwargs, pd.read_csv) + return pd.read_csv(*args, **read_kwargs) + @ParseArgs def read_csv(*args, **kwargs) -> str: read_kwargs = kwargs_in_func(kwargs, pd.read_csv) df = pd.read_csv(*args, **read_kwargs) markdown_kwargs = kwargs_not_in_func(kwargs, pd.read_csv) - return convert_to_md_table(df, markdown_kwargs) + return convert_to_md_table(df, **markdown_kwargs) +@ParseArgs +def pd_read_table(*args, **kwargs) -> str: + read_kwargs = kwargs_in_func(kwargs, pd.read_table) + return pd.read_table(*args, **read_kwargs) + @ParseArgs def read_table(*args, **kwargs) -> str: read_kwargs = kwargs_in_func(kwargs, pd.read_table) df = pd.read_table(*args, **read_kwargs) markdown_kwargs = kwargs_not_in_func(kwargs, pd.read_table) - return convert_to_md_table(df, markdown_kwargs) + return convert_to_md_table(df, **markdown_kwargs) + + +@ParseArgs +def pd_read_fwf(*args, **kwargs) -> str: + read_kwargs = kwargs_in_func(kwargs, pd.read_fwf) + return pd.read_fwf(*args, **read_kwargs) @ParseArgs @@ -84,7 +100,12 @@ def read_fwf(*args, **kwargs) -> str: df = pd.read_fwf(*args, **read_kwargs) markdown_kwargs = kwargs_not_in_func(kwargs, pd.read_fwf) - return convert_to_md_table(df, markdown_kwargs) + return convert_to_md_table(df, **markdown_kwargs) + +@ParseArgs +def pd_read_json(*args, **kwargs) -> str: + read_kwargs = kwargs_in_func(kwargs, pd.read_json) + return pd.read_json(*args, **read_kwargs) @ParseArgs @@ -93,7 +114,12 @@ def read_json(*args, **kwargs) -> str: df = pd.read_json(*args, **read_kwargs) markdown_kwargs = kwargs_not_in_func(kwargs, pd.read_json) - return convert_to_md_table(df, markdown_kwargs) + return convert_to_md_table(df, **markdown_kwargs) + +@ParseArgs +def pd_read_excel(*args, **kwargs) -> str: + read_kwargs = kwargs_in_func(kwargs, pd.read_excel) + return pd.read_excel(*args, **read_kwargs) @ParseArgs @@ -102,9 +128,16 @@ def read_excel(*args, **kwargs) -> str: df = pd.read_excel(*args, **read_kwargs) markdown_kwargs = kwargs_not_in_func(kwargs, pd.read_excel) - return convert_to_md_table(df, markdown_kwargs) + return convert_to_md_table(df, **markdown_kwargs) +@ParseArgs +def pd_read_yaml(*args, **kwargs) -> str: + json_kwargs = kwargs_in_func(kwargs, pd.json_normalize) + with open(args[0], "r") as f: + df = pd.json_normalize(yaml.safe_load(f), **json_kwargs) + return df + @ParseArgs def read_yaml(*args, **kwargs) -> str: json_kwargs = kwargs_in_func(kwargs, pd.json_normalize) @@ -112,7 +145,13 @@ def read_yaml(*args, **kwargs) -> str: df = pd.json_normalize(yaml.safe_load(f), **json_kwargs) markdown_kwargs = kwargs_not_in_func(kwargs, pd.json_normalize) - return convert_to_md_table(df, markdown_kwargs) + return convert_to_md_table(df, **markdown_kwargs) + + +@ParseArgs +def pd_read_feather(*args, **kwargs) -> str: + read_kwargs = kwargs_in_func(kwargs, pd.read_feather) + return pd.read_feather(*args, **read_kwargs) @ParseArgs @@ -121,7 +160,7 @@ def read_feather(*args, **kwargs) -> str: df = pd.read_feather(*args, **read_kwargs) markdown_kwargs = kwargs_not_in_func(kwargs, pd.read_feather) - return convert_to_md_table(df, markdown_kwargs) + return convert_to_md_table(df, **markdown_kwargs) @ParseArgs @@ -145,3 +184,14 @@ def read_raw(*args, **kwargs) -> str: "read_feather": read_feather, "read_raw": read_raw, } + +MACRO_ONLY = { + "pd_read_csv": pd_read_csv, + "pd_read_table": pd_read_table, + "pd_read_fwf": pd_read_fwf, + "pd_read_excel": pd_read_excel, + "pd_read_yaml": pd_read_yaml, + "pd_read_json": pd_read_json, + "pd_read_feather": pd_read_feather, +} +MACROS = {**READERS, **MACRO_ONLY} \ No newline at end of file diff --git a/setup.py b/setup.py index f3fedda..ecd1904 100644 --- a/setup.py +++ b/setup.py @@ -5,7 +5,7 @@ setup( name="mkdocs-table-reader-plugin", - version="3.0.1", + version="3.1.0", description="MkDocs plugin to directly insert tables from files into markdown.", long_description=long_description, long_description_content_type="text/markdown", diff --git a/tests/fixtures/jinja/docs/index.md b/tests/fixtures/jinja/docs/index.md index 82d4f8d..97dc580 100644 --- a/tests/fixtures/jinja/docs/index.md +++ b/tests/fixtures/jinja/docs/index.md @@ -27,3 +27,13 @@ This is a table that we load from the docs folder, because we set `data_path` to {% endfor %} +## Filtering results + +{% raw %} +``` +{{ pd_read_csv("numeric_table.csv").query("a >= 3") | convert_to_md_table }} +``` +{% endraw %} + +{{ pd_read_csv("numeric_table.csv").query("a >= 3") | convert_to_md_table }} + diff --git a/tests/fixtures/jinja/docs/numeric_table.csv b/tests/fixtures/jinja/docs/numeric_table.csv new file mode 100644 index 0000000..598f36c --- /dev/null +++ b/tests/fixtures/jinja/docs/numeric_table.csv @@ -0,0 +1,4 @@ +"a","b" +1,2 +3,4 +5,6 \ No newline at end of file diff --git a/tests/fixtures/search_problem/docs/index.md b/tests/fixtures/search_problem/docs/index.md new file mode 100644 index 0000000..3ce06e2 --- /dev/null +++ b/tests/fixtures/search_problem/docs/index.md @@ -0,0 +1,21 @@ +# Homepage + +```json +{ + "configuration": [ + { + "category": "Advanced", + "component": "NetworkModel", + "defaultvalue": "false", + "description": "This will determine if the admin is allowed to deploy.", + "displaytext": "Admin is allowed to deploy", + "group": "Miscellaneous", + "isdynamic": true, + "name": "deploy.anywhere", + "subgroup": "Others", + "type": "Boolean", + "value": "true" + } + ] +} +``` \ No newline at end of file diff --git a/tests/fixtures/search_problem/mkdocs.yml b/tests/fixtures/search_problem/mkdocs.yml new file mode 100644 index 0000000..fa55d35 --- /dev/null +++ b/tests/fixtures/search_problem/mkdocs.yml @@ -0,0 +1,10 @@ +site_name: test search + +theme: + name: material + features: + - navigation.tabs + +plugins: + - search + - table-reader \ No newline at end of file diff --git a/tests/test_markdown.py b/tests/test_markdown.py index f3a11c1..9425820 100644 --- a/tests/test_markdown.py +++ b/tests/test_markdown.py @@ -14,6 +14,6 @@ def test_convert_to_md_table(): assert df_good.shape[0] > 0 # Because we escape pipes, the 'bad' df - md_bad = convert_to_md_table(df_bad, markdown_kwargs={}) - md_good = convert_to_md_table(df_good, markdown_kwargs={}) + md_bad = convert_to_md_table(df_bad, **{}) + md_good = convert_to_md_table(df_good, **{}) assert md_bad == md_good \ No newline at end of file