diff --git a/ansible_snippets/index.html b/ansible_snippets/index.html index 90e8f4d9b03..3066987d0c7 100644 --- a/ansible_snippets/index.html +++ b/ansible_snippets/index.html @@ -1,4 +1,4 @@ - Ansible Snippets - The Blue Book
Skip to content

Ansible Snippets

Ansible retry a failed job⚑

- command: /usr/bin/false
+ Ansible Snippets - The Blue Book       
  
 
  
 
 
 
 
 
 
  
 
 
 
 
 
  
 
 
 
 
 
Ansible Snippets
 
Ansible lint skip some rules⚑
 
Add a .ansible-lint-ignore file with a line per rule to ignore with the syntax path/to/file rule_to_ignore.
 
Ansible retry a failed job⚑
 
- command: /usr/bin/false
   retries: 3
   delay: 3
   register: result
@@ -199,4 +199,4 @@
 - name: Move foo to bar
   command: mv /path/to/foo /path/to/bar
   when: foo_stat.stat.exists
-
 
 
  Last update: 2023-10-19  
 
 
 
  
  
 
 
 
      
\ No newline at end of file
+
Last update: 2023-11-27
\ No newline at end of file diff --git a/coding/python/mkdocstrings/index.html b/coding/python/mkdocstrings/index.html index ea63e43ed2d..e54c4c0a7ee 100644 --- a/coding/python/mkdocstrings/index.html +++ b/coding/python/mkdocstrings/index.html @@ -35,7 +35,7 @@ With the identifier as title: [full.path.object2][] -

Global options⚑

MkDocstrings accept a few top-level configuration options in mkdocs.yml:

Example:

plugins:
+

Global options⚑

MkDocstrings accept a few top-level configuration options in mkdocs.yml:

Example:

plugins:
 - mkdocstrings:
     default_handler: python
     handlers:
@@ -43,9 +43,9 @@
         rendering:
           show_source: false
     custom_templates: templates
-    watch:
-      - src/my_package
-

The handlers global configuration can then be overridden by local configurations:

::: my_package.my_module.MyClass
+watch:
+  - src/my_package
+

Where watch is a list of directories to watch while serving the documentation. So if any file is changed in those directories, the documentation is rebuilt.

The handlers global configuration can then be overridden by local configurations:

::: my_package.my_module.MyClass
     rendering:
       show_source: true
-

Check the Python handler options for more details.

References⚑

Last update: 2020-12-18
\ No newline at end of file +

Check the Python handler options for more details.

References⚑

Last update: 2023-11-27
\ No newline at end of file diff --git a/emojis/index.html b/emojis/index.html index 3a8e7782835..69f660d23ed 100644 --- a/emojis/index.html +++ b/emojis/index.html @@ -1,4 +1,4 @@ - Emojis - The Blue Book
Skip to content

Emojis

Curated list of emojis to copy paste.

Most used⚑

¯\(°_o)/¯
+ Emojis - The Blue Book       
  
 
  
 
 
 
 
 
 
  
 
 
 
 
 
  
 
 
 
 
 
Emojis
 
Curated list of emojis to copy paste.
 
Most used⚑
 
¯\(°_o)/¯
 
 ¯\_(ツ)_/¯
 
@@ -43,6 +43,7 @@
 
 ＼\ ٩( ᐛ )و /／
 
 
Crying⚑
 
(╥﹏╥)
+(╥_╥)
 (ಥ﹏ಥ)
 
 
Excited⚑
 
(((o(*ﾟ▽ﾟ*)o)))
 
@@ -74,6 +75,7 @@
 
 ( ˘ ³˘)♥
 
 
Love⚑
 
❤
+
 
Lol⚑
 
(*≧▽≦)ﾉｼ))
 
 
Pride⚑
 
<(￣＾￣)>
 
 
Relax⚑
 
_へ__(‾◡◝ )>
 
 
Sad⚑
 
｡ﾟ(*´□`)ﾟ｡
@@ -106,4 +108,4 @@
 
 
WTF⚑
 
(╯°□°)╯ ┻━┻
 
 ヘ（。□°）ヘ
-
   
 
  Last update: 2023-11-10  
 
 
 
  
  
 
 
 
      
\ No newline at end of file
+
Last update: 2023-11-27
\ No newline at end of file diff --git a/gtd/index.html b/gtd/index.html index bbbf547e45f..076eb1f4ee2 100644 --- a/gtd/index.html +++ b/gtd/index.html @@ -1,4 +1,4 @@ - Getting Things Done - The Blue Book
Skip to content

Getting Things Done

icij/etting things done](https://en.wikipedia.org/wiki/Getting_Things_Done), commonly known as GTD is a task management method created by David Allen and published in a book of the same name.⚑

The GTD method rests on the idea of moving all items of interest, relevant information, issues, tasks and projects out of one's mind by recording them externally and then breaking them into actionable work items with known time limits. This allows one's attention to focus on taking action on each task listed in an external record, instead of recalling them intuitively.

The method was described by David Allen in a book with the same name. It's clear that the book is the corner stone of David's business. He is selling his method on every word, some times to the point of tiresome. It's also repeats the same ideas on different parts of the book, I guess that's good in terms of sticking an idea in the people's mind, but if you're already convinced and are trying to sum up the book it's like, hey, I have 90% of the valuable contents of this chapter already in my summary. It's obvious too the context of the writer, that the book was written a while ago and who does it write to. It talks quite often about assistants, bosses of high firm companies he's helped, preferring low-tech physical solutions over digital ones, a lot of references about parenting... If you're able to ignore all the above, it's actually a very good book. The guy has been polishing the method for more than 30 years, and has pretty nice ideas that can change how you manage your life.

My idea of this summary is to try to extract the useful ideas removing all those old-fashioned capitalist values from it.

Theory principles⚑

The theory is based on three key objectives:

  • Capturing all the things that might need to get done or have usefulness for you in a logical and trusted system outside your head and off your mind.
  • Directing yourself to make front-end decisions about all of the “inputs” you let into your life so that you will always have a workable inventory of “next actions” that you can implement or renegotiate in the moment.
  • Curating and coordinating all of that content, utilizing the recognition of the multiple levels of commitments with yourself and others you will have at play, at any point in time.

The idea is to have a system with a coherent set of behaviors and tools that:

  • Functions effectively at the level at which work really happens.
  • Incorporates the results of big-picture thinking as well as the smallest of open details.
  • Manage multiple tiers of priorities.
  • Maintains control over hundreds of new inputs daily.
  • Save a lot more time and effort than are needed to maintain it.
  • Allows you to maximize your time "in the zone" or "flowing".

Managing commitments⚑

Most stress comes from inappropriately managed commitments, every one of them, big or little, is being tracked by a less-than-conscious part of you. These are the “incompletes,” or “open loops,” which are anything pulling at your attention that doesn’t belong where it is, the way it is.

In order to deal effectively with all of that, you must first identify and capture all those things that are “ringing your bell” in some way, clarify what, exactly, they mean to you, and then make a decision about how to move on them.

Managing commitments well requires the implementation of some basic activities and behaviors:

  • If it’s on your mind, your mind isn’t clear. Anything you consider unfinished in any way must be captured in a trusted system outside your mind that you know you’ll come back to regularly and sort through.
  • You must clarify exactly what your commitment is and decide what you have to do, if anything, to make progress toward fulfilling it.
  • Once you’ve decided on all the actions you need to take, you must keep reminders of them organized in a system you review regularly.

These "open loops" make you unreliably think many times on the topic without making any progress. This is a waste of time and energy and only adds to your anxiety about what you should be doing and aren't. There is no reason to ever have the same thought twice, unless you like having that thought.

Managing stuff⚑

"stuff" is anything you have allowed into your psychological or physical world that doesn’t belong where it is, but for which you haven’t yet determined what, exactly, it means to you, with the desired outcome and the next action step. The reason most organizing systems haven’t worked for most people is that they haven’t yet transformed all the stuff they’re trying to organize. As long as it’s still stuff, it’s not controllable.

Almost all of the to-do lists are merely listings of "stuff", not inventories of the resultant real work that needed to be done. They are partial reminders of a lot of things that are unresolved and as yet untranslated into outcomes and actions. Looking at these often creates more stress than relief, because, though it is a valuable trigger for something that you’ve committed to do or decide something about, it still calls out psychologically, “Decide about me!” And if you do not have the energy or focus at the moment to think and decide, it will simply remind you that you are overwhelmed.

The key to managing all of your stuff is managing your actions.

What you do with your time, what you do with information, and what you do with your body and your focus relative to your priorities. Those are the real options to which you must allocate your limited resources. The substantive issue is how to make appropriate choices about what to do at any point in time. The real work is to manage our actions.

Managing actions⚑

You need to control commitments, projects, and actions in two ways:

  • Horizontally: Horizontal control maintains coherence across all the activities in which you are involved. You need a good system that can keep track of as many of them as possible, supply required information about them on demand, and allow you to shift your focus from one thing to the next quickly and easily.
  • Vertically: Vertical control, in contrast, manages thinking, development, and coordination of individual topics and projects. This is “project planning” in the broad sense. It’s focusing in on a single endeavor, situation, or person and fleshing out whatever ideas, details, priorities, and sequences of events may be required for you to handle it, at least for the moment.

Bottom-Up approach⚑

Intellectually, the most appropriate way ought to be to work from the top down, first uncovering personal and organizational purpose and vision, then defining critical objectives, and finally focusing on the details of implementation. The trouble is, however, that most people are so embroiled in commitments on a day-to-day level that their ability to focus successfully on the larger horizon is seriously impaired. Consequently, a bottom-up approach is usually more effective.

Workflow steps⚑

Capture⚑

It’s important to know what needs to be captured and how to do that most effectively so you can process it appropriately. In order for your mind to let go of the lower-level task of trying to hang on to everything, you have to know that you have truly captured everything that might represent something you have to do or at least decide about, and that at some point in the near future you will process and review all of it.

In order to eliminate “holes in your bucket,” you need to collect and gather placeholders for, or representations of, all the things you consider incomplete in your world: anything personal or professional, big or little, of urgent or minor importance, that you think ought to be different than it currently is and that you have any level of internal commitment to changing.

To manage this inventory of open loops appropriately, you need to capture it into “containers” that hold items in abeyance until you have a few moments to decide what they are and what, if anything, you’re going to do about them. Then you must empty these containers regularly to ensure that they remain viable capture tools.

There are several types of tools, both low and high tech, that can be used to collect your open loops. The following can all serve as versions of an in-tray, capturing self-generated input as well as information from external sources:

  • Physical in-tray
  • Paper-based note-taking devices
  • Digital/audio note-taking devices
  • E-mail and text messaging

You'll probably need to choose a variety of tools to fulfill the different inputs of your life, these should become part of your lifestyle. Keep them close by so no matter where you are you can collect a potentially valuable thought. You should have as many in-trays as you need and as few as you can get by with. If you have too many collection zones, however, you won’t be able to process them easily or consistently.

That's one of the key points of your capture tools, if you don’t empty and process the stuff you’ve collected, your tools aren’t serving any function other than the storage of amorphous material. Emptying the contents does not mean that you have to finish what’s there; it just means that you have to decide more specifically what it is and what should be done with it, and if it’s still unfinished, organize it into your system. You must get it out of the container.

In order to get your “inbox” to empty, however, an integrated life-management system must be in place. Too much stuff is left piled in in-trays (physical and digital) because of a lack of effective systems downstream.

Clarify and organize⚑

This is the component of input management that forms the basis for your personal organisation. It's basically thinking about the item and following the next diagram:

Remember to follow the next rules while processing the items:

  • Process the top item first: that way you treat each element equally, so the "least" important ones are not left dangling forever in your inbox thus thwarting it's purpose.
  • Process one item at a time.
  • Never put anything back into “in.”

For each element you need to ask yourself: "What's the next action?".

If you don't see any then refile the element to:

  • Trash: Throw away, shred, or recycle anything that has no potential future action or reference value. If you leave this stuff mixed in with other categories, it will seriously undermine the system and your clarity in the environment.

  • Someday/Maybe: Here goes the elements that don't have next actions now but may have them in the future. This is the “parking lot” for projects that would be impossible to move on at present but that you don’t want to forget about entirely. You’d like to be reminded of the possibility at regular intervals (probably on your weekly reviews). You can also think this as the backlog. Here also belongs other types of information that only needs to be reviewed when you have an urge to engage in a particular kind of activity such as lists like books to read, web sites to surf, trips to do, things to do, recipes to try... Only if you want to review them often to expand your options for creative exploration, otherwise they belong to the References.

  • References are the rest of information that that requires no action but have intrinsic value as information such as your digital garden or technical books.

Reference systems generally take two forms:

  • Topic and area specific storage:

  • General reference files: Stores ad hoc information that doesn’t belong in some predesigned larger category.

If you can do something about the element, you need to think which is the next physical, visible activity that would be required to move the situation towards closure. It's tricky, something like "set meeting" won't do because it's not descriptive of physical behaviour. There is still stuff to decide how, when, with whom, if you don't do it now you won't empty your head and the uncertainty will create a psychological gap that will make you procrastinate, so define the next action now. "Decide what to do about X" doesn't work either, you may need to gather more information on the topic, but deciding doesn't take time.

Once you have the next action, if it can be done in two minutes or less, do it when you first pick the item up. Even if it is not a high-priority one, do it now if you’re ever going to do it at all. The rationale for the two-minute rule is that it’s more or less the point where it starts taking longer to store and track an item than to deal with it the first time it’s in your hands. Two minutes is just a guideline. If you have a long open window of time in which to process your in-tray, you can extend the cutoff for each item to five or ten minutes. If you’ve got to get to the bottom of all your input rapidly, then you may want to shorten the time to one minute, or even thirty seconds, so you can get through everything a little faster.

There’s nothing you really need to track about your two-minute actions. Just do them. If, however, you take an action and don’t finish the project with that one action, you’ll need to clarify what’s next on it, and manage that according to the same criteria.

If the next action is going to take longer than two minutes, ask yourself, “Am I the best person to be doing it?” If not, hand it off to the appropriate person, in order of priority:

  • Send an e-mail.
  • Write a note or an over-note on paper and route it to that person.
  • Send it a instant message.
  • Add it as an agenda item on a list for your next real-time conversation with that person.
  • Talk with her directly, either face-to-face or by phone.

When you hand it off to someone else, and if you care at all whether something happens as a result, you’ll need to track it. Depending on how active you need to be it can go to your Waiting list or to your tickler.

You need to determine what to do with the rest of the next actions once you do, they will end up in one of the next containers:

I's critical that all of these containers be kept distinct from one another. They each represent a discrete type of agreement we make with ourselves, to be reminded of at a specific time and in a specific way, and if they lose their edges and begin to blend, much of the value of organizing will be lost. That's why capturing and clarifying what your relationship to them is primary to getting organized.

Todo list⚑

This list contains all the next actions and projects you are going to actively work on. Projects are any desired result that can be accomplished within a year that requires more than one action step. This means that some rather small things you might not normally call projects are going to be on your Projects list, as well as some big ones. If one step won’t complete something, some kind of goalpost needs to be set up to remind you that there’s something still left to do. If you don’t have a placeholder to remind you about it, it will slip back into your head. The reason for the one-year time frame is that anything you are committed to finish within that scope needs to be reviewed weekly to feel comfortable about its status. Another way to think of this is as a list of open loops, no matter what the size. This is going to be one of the lists that you'll review more often, and it needs to be manageable, if the items start to grow you may want to track the elements you want to do in the semester, or trimester.

Projects do not initially need to be listed in any particular order, by size, or by priority. They just need to be on a master list so you can review them regularly enough to ensure that appropriate next actions have been defined for each of them. That being said, I like to order them a little bit so that I don't need to read the whole list to choose what to do.

There may be reasons to sort your projects into different subcategories, based upon different areas of your focus, but initially creating a single list of all of them will make it easier to customize your system appropriately as you get more comfortable with its usage. To sort them use tags instead of hierarchical structures, they are more flexible. For example you could use tags for:

  • Context: Where can you do the element: home, computer, mobile, ...
  • Area: Broad categories where the element falls in: activism, caring, self-caring, home, digital services, ...
  • Type: I like to separate the tasks that are meant to survive (maintenance) from the ones that are meant to improve things (improvement)
  • Mood, energy level, time: It's useful to have a quick way to see the tasks you can work on when you don't have that much time (small), you don't have that much mental energy (brainless), when you're sad, ...

For many of your projects, you will accumulate relevant information that you will want to organize by theme or topic or project name. Your Projects list will be merely an index. All of the details, plans, and supporting information that you may need as you work on your various projects should be contained in your References system.

Calendar⚑

The calendar holds reminders of actions you need to take fall into two categories: those about things that have to happen on a specific day or time, and those about things that just need to get done as soon as possible. Your calendar handles the first type of reminder.

These things go on your calendar:

  • Time-Specific actions or appointments.
  • Day-Specific actions: These are things that you need to do sometime on a certain day, but not necessarily at a specific time.
  • Day-Specific information: Information that may be useful on a certain date. This might include directions for appointments, activities that other people will be involved in then, or events of interest. It’s helpful to put short-term tickler information here, too, such as a reminder to call someone after he or she returns from vacation. This is also where you would want to park important reminders about when something might be due, or when something needs to be started, given a determined lead time.

Daily to-do lists don't belong to the calendar because:

  • Constant new input and shifting tactical priorities reconfigure daily work so consistently that it’s virtually impossible to nail down to-do items ahead of time. Having a working game plan as a reference point is always useful, but it must be able to be renegotiated at any moment. Trying to keep a list on the calendar, which must then be reentered on another day if items don’t get done, is demoralizing and a waste of time. The Next Actions lists will hold all of those action reminders, even the most time-sensitive ones. And they won’t have to be rewritten daily.

  • If there’s something on a daily to-do list that doesn’t absolutely have to get done that day, it will dilute the emphasis on the things that truly do. The calendar should be sacred territory. If you write something there, it must get done that day or not at all.

That said, there’s absolutely nothing wrong with creating a quick, informal, short list of “if I have time, I’d really like to . . .” kinds of things, picked from your Next Actions inventory. It just should not be confused with your “have-tos,” and it should be treated lightly enough to discard or change quickly as the inevitable surprises of the day unfold.

Reflect⚑

This is where you take a look at all your outstanding projects and open loops. It’s your chance to scan all the defined actions and options before you, thus radically increasing the efficacy of the choices you make about what you’re doing at any point in time.

The item you’ll probably review most frequently is your calendar, which will remind you about what things truly have to be handled that day. This doesn’t mean that the contents there are the most “important” in some grand sense, only that they must get done. At any point in time, knowing what has to get done and when creates a terrain for maneuvering. It’s a good habit, as soon as you conclude an action on your calendar, to check and see what else remains to be done.

Review whatever lists, overviews, and orientation maps you need to, as often as you need to, to get their contents off your mind.

After checking your calendar, you’ll most often turn to your Next Action lists. These hold the inventory of predefined actions that you can take if you have any discretionary time during the day. If you’ve organized them by context (At Home; At Computer; In Meeting with George) they’ll come into play only when those contexts are available.

Projects, Waiting For, and Someday/Maybe lists need to be reviewed only as often as you think they have to be in order to stop you from wondering about them with a minimum recurrence of once each week.

Engage⚑

The basic purpose of this workflow management process is to facilitate good choices about what you’re doing at any point in time and feel good about it. The idea is to use your intuition on the information you have captured, clarified, organized, and reflected regarding all your current commitments.

First you need to get an awareness of your state. At any point in time you're limited to do the tasks that meet your:

  • Context: Most actions need to be done on a specific location or having some tool at hand. You can use tags to filter the actions available to your context.
  • Time available.
  • Energy available.

Then you need to decide what type of task you want to do:

  • Predefined tasks: Dealing with tasks that you have previously determined need to be done, items of your calendar and Next actions list.
  • Unplanned tasks: Often things come up ad hoc, unsuspected and unforeseen that you either have to or choose to engage in as they occur. When you follow these leads, you’re deciding by default that these things are more important than anything else you have to do at those times.
  • Managing the task system: This entails following the earlier steps of the workflow: capturing stuff, clarifying, organizing and reflecting. As you process your inputs, you’ll no doubt be taking care of some less-than-two-minute actions and tossing and filing numerous things. A good portion of this activity will consist of identifying things that need to get done sometime, but not right away. You’ll be adding to all of your lists as you go along.

And finally decide which one to do based on your priority. Defining priority is a hard task that often involves looking towards the "big picture". The idea is to scale up the abstraction tree below to help you decide:

  • Task level: This is the accumulated list of all the actions you need to take.
  • Project level: Group of tasks that share a relatively short-term outcome you want to achieve.
  • Area level: These are the key areas of your life within which you want to achieve results and maintain standards such as health, social, learning, home, recreation, finances, etc. These are not things to finish but rather to use as criteria for assessing our experiences and our engagements, to maintain balance and sustainability as we live. Listing and reviewing these responsibilities gives a more comprehensive framework for evaluating your inventory of projects.
  • Goal level: What specific achievements you want to meet in the different areas of your life one to two years from now will add another dimension to prioritize your tasks.
  • Vision level: Projecting three to five years into the future generates thinking about bigger categories: organization strategies, environmental trends, career and lifestyle transition circumstances. Internal factors include longer-term career, family, financial, and quality-of-life aspirations and considerations.
  • Purpose and Principles: This is the big-picture view. The primary purpose for anything provides the core definition of what it really is. All goals, visions, objectives, projects, and actions derive from this, and lead toward it.

In real life the important conversations you will have about your focus and your priorities may not exactly fit one level or another. They can provide a useful framework, however, to remind you of the multilayered nature of your commitments and tasks.

Setting priorities in the traditional sense of focusing on your long-term goals and values, though obviously a necessary core focus, does not provide a practical framework for a vast majority of the decisions and tasks you must engage in day to day. Mastering the flow of your work at all the levels you experience that work provides a much more holistic way to get things done and feel good about it.

Setting up the system⚑

To apply the workflow you need to first set the system up, then you'll be able to use and maintain it. To be able to do it save block of time to initialize this process and prepare a workstation with the appropriate space, furniture, and tools. If your space is properly set up and streamlined, it can reduce your unconscious resistance to dealing with your stuff and even make it attractive for you to sit down and crank through your input and your work. An ideal time frame for most people is two whole days, back to back.

Setting up the space⚑

Choose a physical location to serve as as your central cockpit of control. If you already have a desk and office space set up where you work, that’s probably the best place to start. If you work from a home office, obviously that will be your prime location. If you already have both, you’ll want to establish identical, even interchangeable systems in both places, though one will probably be primary.

The basics for a workspace are just a writing surface and room for an in-tray.

Design your system⚑

Before you start moving stuff around it's a good idea to get the first design of your whole system, in my case I'm going to heavily rely on org-mode to track most of the stuff with a repository with the next structure:

.
+ Getting Things Done - The Blue Book       
  
 
  
 
 
 
 
 
 
  
 
 
 
 
 
  
 
 
 
 
 
Getting Things Done
 
Getting things done, commonly known as GTD is a task management method created by David Allen and published in a book of the same name.⚑
 
The GTD method rests on the idea of moving all items of interest, relevant information, issues, tasks and projects out of one's mind by recording them externally and then breaking them into actionable work items with known time limits. This allows one's attention to focus on taking action on each task listed in an external record, instead of recalling them intuitively.
 
The method was described by David Allen in a book with the same name. It's clear that the book is the corner stone of David's business. He is selling his method on every word, some times to the point of tiresome. It's also repeats the same ideas on different parts of the book, I guess that's good in terms of sticking an idea in the people's mind, but if you're already convinced and are trying to sum up the book it's like, hey, I have 90% of the valuable contents of this chapter already in my summary. It's obvious too the context of the writer, that the book was written a while ago and who does it write to. It talks quite often about assistants, bosses of high firm companies he's helped, preferring low-tech physical solutions over digital ones, a lot of references about parenting... If you're able to ignore all the above, it's actually a very good book. The guy has been polishing the method for more than 30 years, and has pretty nice ideas that can change how you manage your life.
 
My idea of this summary is to try to extract the useful ideas removing all those old-fashioned capitalist values from it.
 
Theory principles⚑
 
The theory is based on three key objectives: 
 
     
  • Capturing all the things that might need to get done or have usefulness for you in a logical and trusted system outside your head and off your mind.
    •  
  • Directing yourself to make front-end decisions about all of the “inputs” you let into your life so that you will always have a workable inventory of “next actions” that you can implement or renegotiate in the moment.
    •  
  • Curating and coordinating all of that content, utilizing the recognition of the multiple levels of commitments with yourself and others you will have at play, at any point in time.
    •  
 
The idea is to have a system with a coherent set of behaviors and tools that:
 
     
  • Functions effectively at the level at which work really happens. 
    •  
  • Incorporates the results of big-picture thinking as well as the smallest of open details. 
    •  
  • Manage multiple tiers of priorities. 
    •  
  • Maintains control over hundreds of new inputs daily. 
    •  
  • Save a lot more time and effort than are needed to maintain it. 
    •  
  • Allows you to maximize your time "in the zone" or "flowing".
    •  
 
Managing commitments⚑
 
Most stress comes from inappropriately managed commitments, every one of them, big or little, is being tracked by a less-than-conscious part of you. These are the “incompletes,” or “open loops,” which are anything pulling at your attention that doesn’t belong where it is, the way it is.
 
In order to deal effectively with all of that, you must first identify and capture all those things that are “ringing your bell” in some way, clarify what, exactly, they mean to you, and then make a decision about how to move on them.
 
Managing commitments well requires the implementation of some basic activities and behaviors:
 
     
  • If it’s on your mind, your mind isn’t clear. Anything you consider unfinished in any way must be captured in a trusted system outside your mind that you know you’ll come back to regularly and sort through.
    •  
  • You must clarify exactly what your commitment is and decide what you have to do, if anything, to make progress toward fulfilling it.
    •  
  • Once you’ve decided on all the actions you need to take, you must keep reminders of them organized in a system you review regularly.
    •  
 
These "open loops" make you unreliably think many times on the topic without making any progress. This is a waste of time and energy and only adds to your anxiety about what you should be doing and aren't. There is no reason to ever have the same thought twice, unless you like having that thought.
 
Managing stuff⚑
 
"stuff" is anything you have allowed into your psychological or physical world that doesn’t belong where it is, but for which you haven’t yet determined what, exactly, it means to you, with the desired outcome and the next action step. The reason most organizing systems haven’t worked for most people is that they haven’t yet transformed all the stuff they’re trying to organize. As long as it’s still stuff, it’s not controllable.
 
Almost all of the to-do lists are merely listings of "stuff", not inventories of the resultant real work that needed to be done. They are partial reminders of a lot of things that are unresolved and as yet untranslated into outcomes and actions. Looking at these often creates more stress than relief, because, though it is a valuable trigger for something that you’ve committed to do or decide something about, it still calls out psychologically, “Decide about me!” And if you do not have the energy or focus at the moment to think and decide, it will simply remind you that you are overwhelmed.
 
The key to managing all of your stuff is managing your actions.
 
What you do with your time, what you do with information, and what you do with your body and your focus relative to your priorities. Those are the real options to which you must allocate your limited resources. The substantive issue is how to make appropriate choices about what to do at any point in time. The real work is to manage our actions.
 
Managing actions⚑
 
You need to control commitments, projects, and actions in two ways:
 
     
  • Horizontally: Horizontal control maintains coherence across all the activities in which you are involved. You need a good system that can keep track of as many of them as possible, supply required information about them on demand, and allow you to shift your focus from one thing to the next quickly and easily.
    •  
  • Vertically: Vertical control, in contrast, manages thinking, development, and coordination of individual topics and projects. This is “project planning” in the broad sense. It’s focusing in on a single endeavor, situation, or person and fleshing out whatever ideas, details, priorities, and sequences of events may be required for you to handle it, at least for the moment.
    •  
 
Bottom-Up approach⚑
 
Intellectually, the most appropriate way ought to be to work from the top down, first uncovering personal and organizational purpose and vision, then defining critical objectives, and finally focusing on the details of implementation. The trouble is, however, that most people are so embroiled in commitments on a day-to-day level that their ability to focus successfully on the larger horizon is seriously impaired. Consequently, a bottom-up approach is usually more effective.
 
Workflow steps⚑
  
Capture⚑
 
It’s important to know what needs to be captured and how to do that most effectively so you can process it appropriately. In order for your mind to let go of the lower-level task of trying to hang on to everything, you have to know that you have truly captured everything that might represent something you have to do or at least decide about, and that at some point in the near future you will process and review all of it.
 
In order to eliminate “holes in your bucket,” you need to collect and gather placeholders for, or representations of, all the things you consider incomplete in your world: anything personal or professional, big or little, of urgent or minor importance, that you think ought to be different than it currently is and that you have any level of internal commitment to changing.
 
To manage this inventory of open loops appropriately, you need to capture it into “containers” that hold items in abeyance until you have a few moments to decide what they are and what, if anything, you’re going to do about them. Then you must empty these containers regularly to ensure that they remain viable capture tools.
 
There are several types of tools, both low and high tech, that can be used to collect your open loops. The following can all serve as versions of an in-tray, capturing self-generated input as well as information from external sources:
 
     
  • Physical in-tray
    •  
  • Paper-based note-taking devices
    •  
  • Digital/audio note-taking devices
    •  
  • E-mail and text messaging
    •  
 
You'll probably need to choose a variety of tools to fulfill the different inputs of your life, these should become part of your lifestyle. Keep them close by so no matter where you are you can collect a potentially valuable thought. You should have as many in-trays as you need and as few as you can get by with. If you have too many collection zones, however, you won’t be able to process them easily or consistently.
 
That's one of the key points of your capture tools, if you don’t empty and process the stuff you’ve collected, your tools aren’t serving any function other than the storage of amorphous material. Emptying the contents does not mean that you have to finish what’s there; it just means that you have to decide more specifically what it is and what should be done with it, and if it’s still unfinished, organize it into your system. You must get it out of the container. 
 
In order to get your “inbox” to empty, however, an integrated life-management system must be in place. Too much stuff is left piled in in-trays (physical and digital) because of a lack of effective systems downstream.
 
Clarify and organize⚑
 
This is the component of input management that forms the basis for your personal organisation. It's basically thinking about the item and following the next diagram:
 
 
Remember to follow the next rules while processing the items:
 
     
  • Process the top item first: that way you treat each element equally, so the "least" important ones are not left dangling forever in your inbox thus thwarting it's purpose. 
    •  
  • Process one item at a time.
    •  
  • Never put anything back into “in.”
    •  
 
For each element you need to ask yourself: "What's the next action?".
 
If you don't see any then refile the element to:
 
     
  •  
    Trash: Throw away, shred, or recycle anything that has no potential future action or reference value. If you leave this stuff mixed in with other categories, it will seriously undermine the system and your clarity in the environment.
     
    •  
  •  
    Someday/Maybe: Here goes the elements that don't have next actions now but may have them in the future. This is the “parking lot” for projects that would be impossible to move on at present but that you don’t want to forget about entirely. You’d like to be reminded of the possibility at regular intervals (probably on your weekly reviews). You can also think this as the backlog. Here also belongs other types of information that only needs to be reviewed when you have an urge to engage in a particular kind of activity such as lists like books to read, web sites to surf, trips to do, things to do, recipes to try... Only if you want to review them often to expand your options for creative exploration, otherwise they belong to the References.
     
    •  
  •  
    References are the rest of information that that requires no action but have intrinsic value as information such as your digital garden or technical books.
     
    •  
 
Reference systems generally take two forms: 
 
     
  •  
    Topic and area specific storage: 
     
    •  
  •  
    General reference files: Stores ad hoc information that doesn’t belong in some predesigned larger category. 
     
    •  
 
If you can do something about the element, you need to think which is the next physical, visible activity that would be required to move the situation towards closure. It's tricky, something like "set meeting" won't do because it's not descriptive of physical behaviour. There is still stuff to decide how, when, with whom, if you don't do it now you won't empty your head and the uncertainty will create a psychological gap that will make you procrastinate, so define the next action now. "Decide what to do about X" doesn't work either, you may need to gather more information on the topic, but deciding doesn't take time. 
 
Once you have the next action, if it can be done in two minutes or less, do it when you first pick the item up. Even if it is not a high-priority one, do it now if you’re ever going to do it at all. The rationale for the two-minute rule is that it’s more or less the point where it starts taking longer to store and track an item than to deal with it the first time it’s in your hands. Two minutes is just a guideline. If you have a long open window of time in which to process your in-tray, you can extend the cutoff for each item to five or ten minutes. If you’ve got to get to the bottom of all your input rapidly, then you may want to shorten the time to one minute, or even thirty seconds, so you can get through everything a little faster. 
 
There’s nothing you really need to track about your two-minute actions. Just do them. If, however, you take an action and don’t finish the project with that one action, you’ll need to clarify what’s next on it, and manage that according to the same criteria. 
 
If the next action is going to take longer than two minutes, ask yourself, “Am I the best person to be doing it?” If not, hand it off to the appropriate person, in order of priority:
 
     
  • Send an e-mail.
    •  
  • Write a note or an over-note on paper and route it to that person.
    •  
  • Send it a instant message.
    •  
  • Add it as an agenda item on a list for your next real-time conversation with that person.
    •  
  • Talk with her directly, either face-to-face or by phone.
    •  
 
When you hand it off to someone else, and if you care at all whether something happens as a result, you’ll need to track it. Depending on how active you need to be it can go to your Waiting list or to your tickler.
 
You need to determine what to do with the rest of the next actions once you do, they will end up in one of the next containers:
  
I's critical that all of these containers be kept distinct from one another. They each represent a discrete type of agreement we make with ourselves, to be reminded of at a specific time and in a specific way, and if they lose their edges and begin to blend, much of the value of organizing will be lost. That's why capturing and clarifying what your relationship to them is primary to getting organized.
 
Todo list⚑
 
This list contains all the next actions and projects you are going to actively work on. Projects are any desired result that can be accomplished within a year that requires more than one action step. This means that some rather small things you might not normally call projects are going to be on your Projects list, as well as some big ones. If one step won’t complete something, some kind of goalpost needs to be set up to remind you that there’s something still left to do. If you don’t have a placeholder to remind you about it, it will slip back into your head. The reason for the one-year time frame is that anything you are committed to finish within that scope needs to be reviewed weekly to feel comfortable about its status. Another way to think of this is as a list of open loops, no matter what the size. This is going to be one of the lists that you'll review more often, and it needs to be manageable, if the items start to grow you may want to track the elements you want to do in the semester, or trimester.
 
Projects do not initially need to be listed in any particular order, by size, or by priority. They just need to be on a master list so you can review them regularly enough to ensure that appropriate next actions have been defined for each of them. That being said, I like to order them a little bit so that I don't need to read the whole list to choose what to do.
 
There may be reasons to sort your projects into different subcategories, based upon different areas of your focus, but initially creating a single list of all of them will make it easier to customize your system appropriately as you get more comfortable with its usage. To sort them use tags instead of hierarchical structures, they are more flexible. For example you could use tags for:
 
     
  • Context: Where can you do the element: home, computer, mobile, ...
    •  
  • Area: Broad categories where the element falls in: activism, caring, self-caring, home, digital services, ...
    •  
  • Type: I like to separate the tasks that are meant to survive (maintenance) from the ones that are meant to improve things (improvement)
    •  
  • Mood, energy level, time: It's useful to have a quick way to see the tasks you can work on when you don't have that much time (small), you don't have that much mental energy (brainless), when you're sad, ...
    •  
 
For many of your projects, you will accumulate relevant information that you will want to organize by theme or topic or project name. Your Projects list will be merely an index. All of the details, plans, and supporting information that you may need as you work on your various projects should be contained in your References system.
 
Calendar⚑
 
The calendar holds reminders of actions you need to take fall into two categories: those about things that have to happen on a specific day or time, and those about things that just need to get done as soon as possible. Your calendar handles the first type of reminder.
 
These things go on your calendar:
 
     
  • Time-Specific actions or appointments.
    •  
  • Day-Specific actions: These are things that you need to do sometime on a certain day, but not necessarily at a specific time. 
    •  
  • Day-Specific information: Information that may be useful on a certain date. This might include directions for appointments, activities that other people will be involved in then, or events of interest. It’s helpful to put short-term tickler information here, too, such as a reminder to call someone after he or she returns from vacation. This is also where you would want to park important reminders about when something might be due, or when something needs to be started, given a determined lead time.
    •  
 
Daily to-do lists don't belong to the calendar because:
 
     
  •  
    Constant new input and shifting tactical priorities reconfigure daily work so consistently that it’s virtually impossible to nail down to-do items ahead of time. Having a working game plan as a reference point is always useful, but it must be able to be renegotiated at any moment. Trying to keep a list on the calendar, which must then be reentered on another day if items don’t get done, is demoralizing and a waste of time. The Next Actions lists will hold all of those action reminders, even the most time-sensitive ones. And they won’t have to be rewritten daily.
     
    •  
  •  
    If there’s something on a daily to-do list that doesn’t absolutely have to get done that day, it will dilute the emphasis on the things that truly do. The calendar should be sacred territory. If you write something there, it must get done that day or not at all. 
     
    •  
 
That said, there’s absolutely nothing wrong with creating a quick, informal, short list of “if I have time, I’d really like to . . .” kinds of things, picked from your Next Actions inventory. It just should not be confused with your “have-tos,” and it should be treated lightly enough to discard or change quickly as the inevitable surprises of the day unfold.
 
Reflect⚑
 
This is where you take a look at all your outstanding projects and open loops. It’s your chance to scan all the defined actions and options before you, thus radically increasing the efficacy of the choices you make about what you’re doing at any point in time.
 
The item you’ll probably review most frequently is your calendar, which will remind you about what things truly have to be handled that day. This doesn’t mean that the contents there are the most “important” in some grand sense, only that they must get done. At any point in time, knowing what has to get done and when creates a terrain for maneuvering. It’s a good habit, as soon as you conclude an action on your calendar, to check and see what else remains to be done.
 
Review whatever lists, overviews, and orientation maps you need to, as often as you need to, to get their contents off your mind.
 
After checking your calendar, you’ll most often turn to your Next Action lists. These hold the inventory of predefined actions that you can take if you have any discretionary time during the day. If you’ve organized them by context (At Home; At Computer; In Meeting with George) they’ll come into play only when those contexts are available.
 
Projects, Waiting For, and Someday/Maybe lists need to be reviewed only as often as you think they have to be in order to stop you from wondering about them with a minimum recurrence of once each week.
 
Engage⚑
 
The basic purpose of this workflow management process is to facilitate good choices about what you’re doing at any point in time and feel good about it. The idea is to use your intuition on the information you have captured, clarified, organized, and reflected regarding all your current commitments. 
 
First you need to get an awareness of your state. At any point in time you're limited to do the tasks that meet your:
 
     
  • Context: Most actions need to be done on a specific location or having some tool at hand. You can use tags to filter the actions available to your context.
    •  
  • Time available.
    •  
  • Energy available.
    •  
 
Then you need to decide what type of task you want to do:
 
     
  • Predefined tasks: Dealing with tasks that you have previously determined need to be done, items of your calendar and Next actions list.
    •  
  • Unplanned tasks: Often things come up ad hoc, unsuspected and unforeseen that you either have to or choose to engage in as they occur. When you follow these leads, you’re deciding by default that these things are more important than anything else you have to do at those times.
    •  
  • Managing the task system: This entails following the earlier steps of the workflow: capturing stuff, clarifying, organizing and reflecting. As you process your inputs, you’ll no doubt be taking care of some less-than-two-minute actions and tossing and filing numerous things. A good portion of this activity will consist of identifying things that need to get done sometime, but not right away. You’ll be adding to all of your lists as you go along.
    •  
 
And finally decide which one to do based on your priority. Defining priority is a hard task that often involves looking towards the "big picture". The idea is to scale up the abstraction tree below to help you decide:
 
     
  • Task level: This is the accumulated list of all the actions you need to take.
    •  
  • Project level: Group of tasks that share a relatively short-term outcome you want to achieve.
    •  
  • Area level: These are the key areas of your life within which you want to achieve results and maintain standards such as health, social, learning, home, recreation, finances, etc. These are not things to finish but rather to use as criteria for assessing our experiences and our engagements, to maintain balance and sustainability as we live. Listing and reviewing these responsibilities gives a more comprehensive framework for evaluating your inventory of projects.
    •  
  • Goal level: What specific achievements you want to meet in the different areas of your life one to two years from now will add another dimension to prioritize your tasks. 
    •  
  • Vision level: Projecting three to five years into the future generates thinking about bigger categories: organization strategies, environmental trends, career and lifestyle transition circumstances. Internal factors include longer-term career, family, financial, and quality-of-life aspirations and considerations. 
    •  
  • Purpose and Principles: This is the big-picture view. The primary purpose for anything provides the core definition of what it really is. All goals, visions, objectives, projects, and actions derive from this, and lead toward it.
    •  
 
In real life the important conversations you will have about your focus and your priorities may not exactly fit one level or another. They can provide a useful framework, however, to remind you of the multilayered nature of your commitments and tasks.
 
Setting priorities in the traditional sense of focusing on your long-term goals and values, though obviously a necessary core focus, does not provide a practical framework for a vast majority of the decisions and tasks you must engage in day to day. Mastering the flow of your work at all the levels you experience that work provides a much more holistic way to get things done and feel good about it.
 
Setting up the system⚑
 
To apply the workflow you need to first set the system up, then you'll be able to use and maintain it. To be able to do it save block of time to initialize this process and prepare a workstation with the appropriate space, furniture, and tools. If your space is properly set up and streamlined, it can reduce your unconscious resistance to dealing with your stuff and even make it attractive for you to sit down and crank through your input and your work. An ideal time frame for most people is two whole days, back to back.
 
Setting up the space⚑
 
Choose a physical location to serve as as your central cockpit of control. If you already have a desk and office space set up where you work, that’s probably the best place to start. If you work from a home office, obviously that will be your prime location. If you already have both, you’ll want to establish identical, even interchangeable systems in both places, though one will probably be primary.
 
The basics for a workspace are just a writing surface and room for an in-tray.
 
Design your system⚑
 
Before you start moving stuff around it's a good idea to get the first design of your whole system, in my case I'm going to heavily rely on org-mode to track most of the stuff with a repository with the next structure:
 
.
 ├── calendar
 │   ├── personal.org
 │   │   ├── One time events
@@ -58,4 +58,4 @@
     - [x] Read chapters 6 and 7
     - [ ] Read chapters 8 and 9
   * TODO up the book in the blue book
-
 
That way when Read David Allen's GTD book is done, you can archive it and forget about it. 
 
If the project starts having many subprojects, it may help to have a section "Outcomes" to define what do you want to achieve with the project. It can be accompanied with a "Next Steps" section to add any subproject or action that doesn't match the defined outcomes, and once you finish the project, you can refile them into new projects.
 
The NEXT state⚑
 
It's useful to have a NEXT state to track the first next action you need to deal with for each project. That way when you open the file, you can go to the top of it and search for NEXT and it will lead you directly to where you need to work on.
 
Tag management⚑
 
As explained in the todo list section, you can use tags to filter your tasks. I'm using the next ones:
 
     
  • Area: Broad categories where the element falls in: activism, caring, self-caring, home, digital services, ...
    •  
  • Type: I like to separate the tasks that are meant to survive (maintenance) from the ones that are meant to improve things (improvement). I use these only in the big projects.
    •  
  • :long_break:: I'm using this tag to track the small projects that can be done in the long pomodoro breaks. Depending on the kind of long break that I need I then filter for the next tags:
    •  
  • brainless: If I want to keep on thinking on what I was doing, an example could be emptying the dishwasher, watering the plants, ...
    •  
  • call: If I want to completely change context and want some social interaction. For example call mom. 
    •  
  • :thinking:: Used to track the elements where you just need to think about them. For example I like to have this list to have a prioritized list to deal with when I'm in the shower, while biking, hiking...
    •  
  • :relax:: Used to track the things you can do when you just want to chill: really listen the music of a group, summarize a book, clean your music library...
    •  
  • People involved: :marie:, :billy:, ...
    •  
 
Always use lowercase tags, it will save you some key strokes.
 
Priority management⚑
 
You shouldn’t bother to create some external structuring of the priorities on your lists that you’ll then have to rearrange or rewrite as things change. Attempting to impose such scaffolding has been a big source of frustration in many people’s organizing. You’ll be prioritizing more intuitively as you see the whole list against quite a number of shifting variables. The list is just a way for you to keep track of the total inventory of active things to which you have made a commitment, and to have that inventory available for review.
 
Therefore I'm going to try not to use orgmode's priorities for the tasks.
 
Waiting tasks⚑
 
Waiting actions are elements that are blocked for any reason. I use the WAITING TODO keyword to track this state. Under each element you should add that reason and optionally the process you want to follow to unblock it.
 
If you need to actively track the evolution of the WAITING status, leave it on the top of your todo. Otherwise set the date you want to check its status and move it to the ticker.org file. If you don't even want to track it, move it to the someday.org file.
 
Tickler management⚑
 
The tickler is a system where you postpone actions to a specific date, but not with a calendar mindset where the action needs to be done at that date. With the tickler you schedule the action to enter your inbox that day to decide what are you going to do with it.
 
To implement this in orgmode you can add the :tickler: tag to any element that is tracked in the agenda files and once a day you can look at the day's agenda and decide what to do with the action. It's important though that whatever you do with it, you have to remove it from the agenda view in order to only keep the elements that you need to do in the day. You can follow this workflow by:
 
     
  • Opening the agenda view gaa 
    •  
  • Go to the view of the day vd
    •  
  • Go to today .
    •  
  • Search by tickler /tickler
    •  
 
It can also help to review in the weeklies all the ticklers of the week to avoid surprises.
 
If you want to make the project go away from your todo or someday until the tickler date, move it to the tickler.org file.
 
Soft recurrent tasks⚑
 
There are some tasks that have a soft recurrence meaning that once you do them you don't want them to show up in your list of actions until a specific time has passed. You could use a recurrent DEADLINE or SCHEDULED but as we've seen earlier that will clutter your calendar pretty soon. Try following the next workflow with these tasks:
 
     
  • Add the :soft_recurrence: tag to keep them tracked.
    •  
  • Add them to the tickler file with a recurrent appointment date <2023-03-13 Mon ++1w> and the :tickler: tag so that it doesn't show up in the agenda view even if you move it to another file.
    •  
  • When the appointed day comes you'll review the tickler elements as part of your day's routine. If you think it's time to do it, refile it to the todo.org file, if not, adjust the recurrence period and set the next date. Even though this workflow is reproducing the "kick the can forward" that we want to avoid, the idea is that once you get the period right you'll never have to do it again. If you see that after some iterations the period keeps on changing, maybe this workflow is not working for that kind of task and you may need to think of a better system ¯\(°_o)/¯.
    •  
  • Once you complete the item, the new one will be spawned, once it has refile it to the tickler file again.
    •  
 
We use appointments instead of DEADLINE or SCHEDULED so that they don't clutter the tickler view if you don't do them on the appointment date. 
 
Another option is not to archive the DONE tasks and in the weekly reset them to TODO the ones that you want to do the next week.
 
Setting up your filing system⚑
 
You will resist the whole process of capturing information if your reference systems are not fast, functional, and fun. Thus you need to envision a system at hand that supports both physical and digital content. Without a streamlined system for both, you will resist keeping potentially valuable information, or what you do keep will accumulate in inappropriate places. 
 
We’re concerned here mostly with general-reference filing, anything that you want to keep for its interesting or useful data or purpose and that doesn’t fit into your specialized filing systems and won’t stand up by itself on a shelf. For example articles, brochures, pieces of paper, notes, printouts, documents, and even physical things like tickets, keys, buyers-club membership cards, and flash drives.
 
It should take you less than one minute to pick something up out of your in-tray, decide it needs no next action but has some potential future value, and finish storing it in a trusted system. If it takes longer, you’ll likely stack it or stuff it somewhere instead. Besides being fast, the system needs to be fun and easy, current and complete. Otherwise you’ll unconsciously resist emptying your in-tray because you know there’s likely to be something in there that ought to get filed, and you won’t even want to look at the papers or your clogged e-mail. If you have to get up every time you have some ad hoc piece of paper you want to file, or you have to search multiple places on your computer for an appropriate location for a piece of information you want to keep, you’ll tend to stack it or leave it in its original place instead of filing it.
 
You must feel equally comfortable about filing a single piece of paper on a new topic, even a scribbled note, in its own file as you would about filing a more formal, larger document.
 
Whatever you need to do to get your reference system to that quick and easy standard for everything it has to hold, do it. For example purge your files at least once a year, that keeps it from going stale and becoming the dreaded black hole- .
 
Digital general reference⚑
 
It is very helpful to have a visual map sorted in ways that make sense, either by indexes or data groups organized effectively, usually in an alphabetic order.
 
The biggest issue for digitally oriented people is that the ease of capturing and storing has generated a write-only syndrome: all they’re doing is capturing information—not actually accessing and using it intelligently. Some consciousness needs to be applied to keep one’s potentially huge digital library functional, versus a black hole of data easily dumped in there with a couple of keystrokes.
 
You need to consistently check how much room to give yourself so that the content remains meaningfully and easily accessible, without creating a black hole of an inordinate amount of information amorphously organized. 
 
Physical general reference⚑
 
One idea is to have one system/place where you order the content alphabetically, not multiple ones. People have a tendency to want to use their files as a personal management system, and therefore they attempt to organize them in groupings by projects or areas of focus. This magnifies geometrically the number of places something isn’t when you forget where you filed it.
 
Capture all your stuff⚑
 
The focus of this process is to capture everything that has your attention, otherwise some part of you will still not totally trust that you're working with the whole picture. While you're doing it, create a list of all the sources of inputs in your world. 
 
What you're going to do is methodically go through each piece of your life and search for anything that doesn’t permanently belong where it is, the way it is, and put it into your in-tray. You’ll be gathering things that are incomplete or things that have some decision about potential action tied to them. They all go into your “inbox”, so they’ll be available for later processing. If it's immediately evident that you don't need the stuff trash it.
 
Be patient, this process may take between 1 and 6 hours, and at the end you'll have a huge pile of stuff in your inbox. You might be scared and get the thought of "what am I doing with my life?", but don't worry you'll get everything in order soon :).
 
Define what is going to be your in-tray⚑
 
To be able to store all the "stuff" that needs to be dealt with you need to define what is your in-tray going to be, some suggestions are:
 
     
  • A clear room: Make some space somewhere at the location you're going to do the gathering to pile physical stuff
    •  
  • A physical or digital list.
    •  
 
Physical gathering⚑
 
The first activity is to search your physical environment. The best way to create a clean decision about whether something should go into the in-tray is to understand clearly what shouldn’t go in. Here are the four categories of things that can remain where they are, the way they are, with no action tied to them:
 
     
  • Supplies: Anything you need to keep because you use it regularly. 
    •  
  • Reference Material: Anything you simply keep for information as needed. This category includes manuals, all your telephone and address information, any material relevant to projects, themes, and topics. It also includes books and magazines that you may be keeping as a library.
    •  
  • Decoration.
    •  
  • Equipment: Your phone, computer, printer, wastebasket, furniture, clock, chargers, pens, and notepads.
    •  
 
Everything else goes into your inbox. But many of the things you might initially interpret as supplies, reference, decoration, or equipment could also have action associated with them because they still aren’t exactly the way they need to be.
 
As you engage in the capturing step, you may run into one or more of the following problems:
 
     
  • An item is too big to go in the in-tray: create a post it that represents it or add it as an entry in your digital inbox. If you can, add the date too
    •  
  • The pile is too big to fit the in-tray: Create visually distinct stacks around the in-tray, even on the floor.
    •  
  • Doubts whether to trash something: When in doubt keep it, you'll decide about it later when you process the in-tray. What you need to avoid is to get caught up in deciding what to do with the element. That's going to be the next step in the process, let's go one at a time.
    •  
  • Getting caught up in cleaning and organizing: If it doesn't take that long it's fine but remember the purpose of this process and the fact that we want to finish it as soon as possible. If you discover things you want to change, add them to the in-tray.
    •  
  • If you encounter stuff that is already on lists and organizers, treat them as everything else in the "in".
    •  
 
Now that the process it's clear let's start.
 
Start with the space where you actually do stuff, scan the place centimeter by centimeter with the mindset defined above, check your desk, drawers, floors, walls, shelves, equipment, furniture, fixtures...Then repeat the process with each room of your home.
 
Mental gathering⚑
 
Once you already have a nice pile of stuff, think of what has your attention that isn’t represented by something already in your in-tray and record each thought, each idea, each project or thing that occurs you and add it to the inbox.
 
To assist in clearing your head, you may want to review the following the next trigger list, item by item, to see if you’ve forgotten anything. 
 
Personal⚑
 
     
  • Projects started, not completed
    •  
  • Projects that need to be started
    •  
  • Projects—other organizations
    •  
  • Activism
    •  
  • Community
    •  
  • Volunteer
    •  
  • Spiritual organization
    •  
  • Commitments/promises to others
    •  
  • Friends
    •  
  • Partner
    •  
  • Family
    •  
  • Parents
    •  
  • Children
    •  
  • Professionals
    •  
  • Returnable items
    •  
  • Debts
    •  
  • Communications to make/get
    •  
  • Calls
    •  
  • Instant messages
    •  
  • Voice messages
    •  
  • E-mails
    •  
  • Cards and letters
    •  
  • Thank-yous
    •  
  • Texts
    •  
  • Social media postings
    •  
  • Upcoming events
    •  
  • Birthdays
    •  
  • Anniversaries
    •  
  • Holidays
    •  
  • Vacation
    •  
  • Travel
    •  
  • Dinners
    •  
  • Parties
    •  
  • Cultural events
    •  
  • Sporting events
    •  
  • Weddings
    •  
  • Graduations
    •  
  • Receptions
    •  
  • Outings
    •  
  • Administration
    •  
  • Home office supplies
    •  
  • Equipment
    •  
  • Phones
    •  
  • Mobile devices
    •  
  • Audio/video media
    •  
  • Voice mail
    •  
  • Computers
    •  
  • Software
    •  
  • Internet
    •  
  • Filing and records
    •  
  • Data storage/backup
    •  
  • Health
    •  
  • Public health system
    •  
  • Doctors
    •  
  • Dentist
    •  
  • Optometrist
    •  
  • Healthcare specialists
    •  
  • Checkups
    •  
  • Diet
    •  
  • Food
    •  
  • Exercise
    •  
  • Leisure
    •  
  • Books
    •  
  • Music
    •  
  • Video
    •  
  • Movies
    •  
  • TV shows
    •  
  • Hiking routes
    •  
  • Travel
    •  
  • Places to visit
    •  
  • People to visit
    •  
  • Web browsing
    •  
  • Photography
    •  
  • Sports equipment
    •  
  • Hobbies
    •  
  • Cooking
    •  
  • Recreation
    •  
  • Errands
    •  
  • Shopping
    •  
  • Stores
    •  
  • Hardware
    •  
  • Supplies
    •  
  • Groceries
    •  
  • Gifts
    •  
  • Pharmacy
    •  
  • Bank
    •  
  • Cleaners
    •  
  • Repairs
    •  
  • Financial
    •  
  • Worries
    •  
  • Taxes
    •  
  • Bills
    •  
  • Banks
    •  
  • Loans
    •  
  • Budget
    •  
  • Insurance
    •  
  • Mortgage
    •  
  • Bookkeeping
    •  
  • Investments
    •  
  • Accountants
    •  
  • Pets
    •  
  • Health
    •  
  • Supplies
    •  
  • Training
    •  
  • Legal
    •  
  • Wills
    •  
  • Trusts
    •  
  • Estate
    •  
  • Legal affairs
    •  
  • Friend/Family projects/activities
    •  
  • Friends
    •  
  • Partner
    •  
  • Parents
    •  
  • Relatives
    •  
  • Children
    •  
  • Home/household
    •  
  • Worries
    •  
  • Rent
    •  
  • Real estate
    •  
  • Repairs
    •  
  • Construction
    •  
  • Remodeling
    •  
  • Landlords
    •  
  • Heating and air conditioning
    •  
  • Plumbing
    •  
  • Utilities
    •  
  • Roof
    •  
  • Landscaping
    •  
  • Driveway
    •  
  • Garage
    •  
  • Walls
    •  
  • Floors
    •  
  • Ceilings
    •  
  • Decor
    •  
  • Furniture
    •  
  • Appliances
    •  
  • Lights and wiring
    •  
  • Kitchen supplies/equipment
    •  
  • Laundry
    •  
  • Purging, organizing, cleaning
    •  
  • Storage
    •  
  • Service providers
    •  
  • Personal development
    •  
  • Classes
    •  
  • Seminars
    •  
  • Education
    •  
  • Coaching/counseling
    •  
  • Career
    •  
  • Creative expressions
    •  
  • Transportation
    •  
  • Bicycles
    •  
  • Maintenance
    •  
  • Repair
    •  
  • Commuting
    •  
  • Motor vehicles
    •  
  • Clothes
    •  
  • Casual
    •  
  • Formal
    •  
  • Sports
    •  
  • Accessories
    •  
  • Luggage
    •  
  • Repairs
    •  
  • Tailoring
    •  
  • Professional
    •  
  • Community
    •  
  • Activism
    •  
  • Neighborhood
    •  
  • Neighbors
    •  
  • Voting
    •  
  • Waiting for
    •  
  • Product orders
    •  
  • Repairs
    •  
  • Reimbursements
    •  
  • Loaned items
    •  
  • Information
    •  
  • Projects/tasks completed by family/friends
    •  
 
Professional⚑
 
     
  • Projects started, not completed
    •  
  • Projects that need to be started
    •  
  • “Look into . . .” projects
    •  
  • Commitments/promises to others
    •  
  • Colleagues
    •  
  • Boss/partners
    •  
  • Others in organization
    •  
  • “Outside” people
       
    • Customers
      •  
    • Other organizations
      •  
    • Professionals
      •  
    • Vendors
      •  
     
    •  
  • Communications to make/get
    •  
  • Internal/external
    •  
  • Initiate or respond to:
       
    • Phone calls
      •  
    • Voice notes
      •  
    • E-mails
      •  
    • Text messages
      •  
    • Letters
      •  
    • Social media postings
      •  
     
    •  
  • Other writing to finish/submit
    •  
  • Reports
    •  
  • Evaluations/reviews
    •  
  • Proposals
    •  
  • Articles
    •  
  • Manuals/instructions
    •  
  • Summaries
    •  
  • Rewrites and edits
    •  
  • Status reporting
    •  
  • Conversation and communication tracking
    •  
  • Meetings that need to be set/requested
    •  
  • Who needs to know about what decisions?
    •  
  • Significant read/review
    •  
  • Professional development
    •  
  • Training/seminars
    •  
  • Things to learn
    •  
  • Things to find out
    •  
  • Skills to practice/develop
    •  
  • Books to read/study
    •  
  • Research
    •  
  • Formal education (licensing, degrees)
    •  
  • Career research
    •  
  • Résumé
    •  
  • Performance objectives
    •  
  • Financial
    •  
  • Forecasts/projections
    •  
  • Credit line
    •  
  • Stability
    •  
  • Planning/organizing
    •  
  • Formal planning (goals, targets, objectives)
    •  
  • Current projects (next stages)
    •  
  • Organizational initiatives
    •  
  • Upcoming events
    •  
  • Meetings
    •  
  • Presentations
    •  
  • Conferences
    •  
  • Organizational structuring
    •  
  • Changes in facilities
    •  
  • Installation of new systems/equipment
    •  
  • Travel
    •  
  • Vacation
    •  
  • Business trips
    •  
  • Waiting for . . .
    •  
  • Information
    •  
  • Delegated tasks/projects
    •  
  • Completions critical to projects
    •  
  • Answers to questions
    •  
  • Replies to:
       
    • E-mails
      •  
    • Letters
      •  
    • Proposals
      •  
    • Calls
      •  
    • Invitations
      •  
     
    •  
  • Requisitions
    •  
  • Reimbursements
    •  
  • Insurance claims
    •  
  • Ordered items
    •  
  • Repairs
    •  
  • Tickets
    •  
  • Decisions of others
    •  
  • Organization Development
    •  
  • Organization chart
    •  
  • Restructuring
    •  
  • Roles
    •  
  • Job descriptions
    •  
  • Facilities
    •  
  • New systems
    •  
  • Leadership
    •  
  • Change initiatives
    •  
  • Succession planning
    •  
  • Organization culture
    •  
  • Administration
    •  
  • Legal
    •  
  • Insurance
    •  
  • Personnel
    •  
  • Staffing
    •  
  • Policies/procedures
    •  
  • Training
    •  
  • Staff
    •  
  • Hiring/firing/promoting
    •  
  • Reviews
    •  
  • Communication
    •  
  • Staff development
    •  
  • Compensation
    •  
  • Feedback
    •  
  • Morale
    •  
  • Systems
    •  
  • Mobile devices
    •  
  • Phones
    •  
  • Computers
    •  
  • Software
    •  
  • Databases
    •  
  • Telecommunications
    •  
  • Internet
    •  
  • Filing and reference
    •  
  • Inventories
    •  
  • Storage
    •  
  • Office/site
    •  
  • Space/arrangements
    •  
  • Furniture
    •  
  • Equipment
    •  
  • Decorations
    •  
  • Utilities
    •  
  • Supplies
    •  
  • Maintenance/cleaning
    •  
  • Security
    •  
 
Now that you know where do your inputs come from you need to think how do you want to manage them from now on to ensure that you're able to be able to continuously capture items in a frictionless way.
 
Empty the inbox⚑
 
Now that we have collected everything that has your attention, you need to get to the bottom of your inbox. To be able to do it in a reasonable amount of time you are not meant to actually do the items themselves, instead analyze each item and decide what it is, what it means and what are you going to do with it.
 
Follow the steps of Clarify and organize until you've reached the bottom of your inbox. You’ll dump a mess of things, file a bunch, do a lot of two-minute actions, and hand off a number of items to other people. You’ll also wind up with a stack of items that have actions associated with them that you still need to do soon, someday, or on a specific date, and reminders of things you’re waiting on from other people. Now that you have everything at the next action level, we need to scale up in the abstraction ladder so that we can prioritize better what to do next.
 
Unclassified thoughts⚑
 
     
  • There must be zero resistance to using the systems we have. Having to continually reinvent our in-tray, our filing system, and how and where we process our stuff can only be a source of incessant distraction.
    •  
  • One of the best tricks for enhancing your productivity is having organizing tools you love to use.
    •  
  • Being organized means nothing more or less than where something is matches what it means to you.
    •  
  • Your organisation system is not something that you'll create all at once. It will evolve as you process yuor stuff and test out whether you have put everything in the best place for you. It won't remain static, it will evolve as you do.
    •  
 
The weekly review⚑
 
The Weekly Review is the time to:
 
     
  • Gather and process all your stuff.
    •  
  • Review your system.
    •  
  • Update your lists.
    •  
  • Get clean, clear, current, and complete.
    •  
 
 
  Last update: 2023-10-19  
 
 
 
  
  
 
 
 
      
\ No newline at end of file
+

That way when Read David Allen's GTD book is done, you can archive it and forget about it.

If the project starts having many subprojects, it may help to have a section "Outcomes" to define what do you want to achieve with the project. It can be accompanied with a "Next Steps" section to add any subproject or action that doesn't match the defined outcomes, and once you finish the project, you can refile them into new projects.

The NEXT state⚑

It's useful to have a NEXT state to track the first next action you need to deal with for each project. That way when you open the file, you can go to the top of it and search for NEXT and it will lead you directly to where you need to work on.

Tag management⚑

As explained in the todo list section, you can use tags to filter your tasks. I'm using the next ones:

  • Area: Broad categories where the element falls in: activism, caring, self-caring, home, digital services, ...
  • Type: I like to separate the tasks that are meant to survive (maintenance) from the ones that are meant to improve things (improvement). I use these only in the big projects.
  • :long_break:: I'm using this tag to track the small projects that can be done in the long pomodoro breaks. Depending on the kind of long break that I need I then filter for the next tags:
  • brainless: If I want to keep on thinking on what I was doing, an example could be emptying the dishwasher, watering the plants, ...
  • call: If I want to completely change context and want some social interaction. For example call mom.
  • :thinking:: Used to track the elements where you just need to think about them. For example I like to have this list to have a prioritized list to deal with when I'm in the shower, while biking, hiking...
  • :relax:: Used to track the things you can do when you just want to chill: really listen the music of a group, summarize a book, clean your music library...
  • People involved: :marie:, :billy:, ...

Always use lowercase tags, it will save you some key strokes.

Priority management⚑

You shouldn’t bother to create some external structuring of the priorities on your lists that you’ll then have to rearrange or rewrite as things change. Attempting to impose such scaffolding has been a big source of frustration in many people’s organizing. You’ll be prioritizing more intuitively as you see the whole list against quite a number of shifting variables. The list is just a way for you to keep track of the total inventory of active things to which you have made a commitment, and to have that inventory available for review.

Therefore I'm going to try not to use orgmode's priorities for the tasks.

Waiting tasks⚑

Waiting actions are elements that are blocked for any reason. I use the WAITING TODO keyword to track this state. Under each element you should add that reason and optionally the process you want to follow to unblock it.

If you need to actively track the evolution of the WAITING status, leave it on the top of your todo. Otherwise set the date you want to check its status and move it to the ticker.org file. If you don't even want to track it, move it to the someday.org file.

Tickler management⚑

The tickler is a system where you postpone actions to a specific date, but not with a calendar mindset where the action needs to be done at that date. With the tickler you schedule the action to enter your inbox that day to decide what are you going to do with it.

To implement this in orgmode you can add the :tickler: tag to any element that is tracked in the agenda files and once a day you can look at the day's agenda and decide what to do with the action. It's important though that whatever you do with it, you have to remove it from the agenda view in order to only keep the elements that you need to do in the day. You can follow this workflow by:

  • Opening the agenda view gaa
  • Go to the view of the day vd
  • Go to today .
  • Search by tickler /tickler

It can also help to review in the weeklies all the ticklers of the week to avoid surprises.

If you want to make the project go away from your todo or someday until the tickler date, move it to the tickler.org file.

Soft recurrent tasks⚑

There are some tasks that have a soft recurrence meaning that once you do them you don't want them to show up in your list of actions until a specific time has passed. You could use a recurrent DEADLINE or SCHEDULED but as we've seen earlier that will clutter your calendar pretty soon. Try following the next workflow with these tasks:

  • Add the :soft_recurrence: tag to keep them tracked.
  • Add them to the tickler file with a recurrent appointment date <2023-03-13 Mon ++1w> and the :tickler: tag so that it doesn't show up in the agenda view even if you move it to another file.
  • When the appointed day comes you'll review the tickler elements as part of your day's routine. If you think it's time to do it, refile it to the todo.org file, if not, adjust the recurrence period and set the next date. Even though this workflow is reproducing the "kick the can forward" that we want to avoid, the idea is that once you get the period right you'll never have to do it again. If you see that after some iterations the period keeps on changing, maybe this workflow is not working for that kind of task and you may need to think of a better system ¯\(°_o)/¯.
  • Once you complete the item, the new one will be spawned, once it has refile it to the tickler file again.

We use appointments instead of DEADLINE or SCHEDULED so that they don't clutter the tickler view if you don't do them on the appointment date.

Another option is not to archive the DONE tasks and in the weekly reset them to TODO the ones that you want to do the next week.

Setting up your filing system⚑

You will resist the whole process of capturing information if your reference systems are not fast, functional, and fun. Thus you need to envision a system at hand that supports both physical and digital content. Without a streamlined system for both, you will resist keeping potentially valuable information, or what you do keep will accumulate in inappropriate places.

We’re concerned here mostly with general-reference filing, anything that you want to keep for its interesting or useful data or purpose and that doesn’t fit into your specialized filing systems and won’t stand up by itself on a shelf. For example articles, brochures, pieces of paper, notes, printouts, documents, and even physical things like tickets, keys, buyers-club membership cards, and flash drives.

It should take you less than one minute to pick something up out of your in-tray, decide it needs no next action but has some potential future value, and finish storing it in a trusted system. If it takes longer, you’ll likely stack it or stuff it somewhere instead. Besides being fast, the system needs to be fun and easy, current and complete. Otherwise you’ll unconsciously resist emptying your in-tray because you know there’s likely to be something in there that ought to get filed, and you won’t even want to look at the papers or your clogged e-mail. If you have to get up every time you have some ad hoc piece of paper you want to file, or you have to search multiple places on your computer for an appropriate location for a piece of information you want to keep, you’ll tend to stack it or leave it in its original place instead of filing it.

You must feel equally comfortable about filing a single piece of paper on a new topic, even a scribbled note, in its own file as you would about filing a more formal, larger document.

Whatever you need to do to get your reference system to that quick and easy standard for everything it has to hold, do it. For example purge your files at least once a year, that keeps it from going stale and becoming the dreaded black hole- .

Digital general reference⚑

It is very helpful to have a visual map sorted in ways that make sense, either by indexes or data groups organized effectively, usually in an alphabetic order.

The biggest issue for digitally oriented people is that the ease of capturing and storing has generated a write-only syndrome: all they’re doing is capturing information—not actually accessing and using it intelligently. Some consciousness needs to be applied to keep one’s potentially huge digital library functional, versus a black hole of data easily dumped in there with a couple of keystrokes.

You need to consistently check how much room to give yourself so that the content remains meaningfully and easily accessible, without creating a black hole of an inordinate amount of information amorphously organized.

Physical general reference⚑

One idea is to have one system/place where you order the content alphabetically, not multiple ones. People have a tendency to want to use their files as a personal management system, and therefore they attempt to organize them in groupings by projects or areas of focus. This magnifies geometrically the number of places something isn’t when you forget where you filed it.

Capture all your stuff⚑

The focus of this process is to capture everything that has your attention, otherwise some part of you will still not totally trust that you're working with the whole picture. While you're doing it, create a list of all the sources of inputs in your world.

What you're going to do is methodically go through each piece of your life and search for anything that doesn’t permanently belong where it is, the way it is, and put it into your in-tray. You’ll be gathering things that are incomplete or things that have some decision about potential action tied to them. They all go into your “inbox”, so they’ll be available for later processing. If it's immediately evident that you don't need the stuff trash it.

Be patient, this process may take between 1 and 6 hours, and at the end you'll have a huge pile of stuff in your inbox. You might be scared and get the thought of "what am I doing with my life?", but don't worry you'll get everything in order soon :).

Define what is going to be your in-tray⚑

To be able to store all the "stuff" that needs to be dealt with you need to define what is your in-tray going to be, some suggestions are:

  • A clear room: Make some space somewhere at the location you're going to do the gathering to pile physical stuff
  • A physical or digital list.

Physical gathering⚑

The first activity is to search your physical environment. The best way to create a clean decision about whether something should go into the in-tray is to understand clearly what shouldn’t go in. Here are the four categories of things that can remain where they are, the way they are, with no action tied to them:

  • Supplies: Anything you need to keep because you use it regularly.
  • Reference Material: Anything you simply keep for information as needed. This category includes manuals, all your telephone and address information, any material relevant to projects, themes, and topics. It also includes books and magazines that you may be keeping as a library.
  • Decoration.
  • Equipment: Your phone, computer, printer, wastebasket, furniture, clock, chargers, pens, and notepads.

Everything else goes into your inbox. But many of the things you might initially interpret as supplies, reference, decoration, or equipment could also have action associated with them because they still aren’t exactly the way they need to be.

As you engage in the capturing step, you may run into one or more of the following problems:

  • An item is too big to go in the in-tray: create a post it that represents it or add it as an entry in your digital inbox. If you can, add the date too
  • The pile is too big to fit the in-tray: Create visually distinct stacks around the in-tray, even on the floor.
  • Doubts whether to trash something: When in doubt keep it, you'll decide about it later when you process the in-tray. What you need to avoid is to get caught up in deciding what to do with the element. That's going to be the next step in the process, let's go one at a time.
  • Getting caught up in cleaning and organizing: If it doesn't take that long it's fine but remember the purpose of this process and the fact that we want to finish it as soon as possible. If you discover things you want to change, add them to the in-tray.
  • If you encounter stuff that is already on lists and organizers, treat them as everything else in the "in".

Now that the process it's clear let's start.

Start with the space where you actually do stuff, scan the place centimeter by centimeter with the mindset defined above, check your desk, drawers, floors, walls, shelves, equipment, furniture, fixtures...Then repeat the process with each room of your home.

Mental gathering⚑

Once you already have a nice pile of stuff, think of what has your attention that isn’t represented by something already in your in-tray and record each thought, each idea, each project or thing that occurs you and add it to the inbox.

To assist in clearing your head, you may want to review the following the next trigger list, item by item, to see if you’ve forgotten anything.

Personal⚑
  • Projects started, not completed
  • Projects that need to be started
  • Projects—other organizations
  • Activism
  • Community
  • Volunteer
  • Spiritual organization
  • Commitments/promises to others
  • Friends
  • Partner
  • Family
  • Parents
  • Children
  • Professionals
  • Returnable items
  • Debts
  • Communications to make/get
  • Calls
  • Instant messages
  • Voice messages
  • E-mails
  • Cards and letters
  • Thank-yous
  • Texts
  • Social media postings
  • Upcoming events
  • Birthdays
  • Anniversaries
  • Holidays
  • Vacation
  • Travel
  • Dinners
  • Parties
  • Cultural events
  • Sporting events
  • Weddings
  • Graduations
  • Receptions
  • Outings
  • Administration
  • Home office supplies
  • Equipment
  • Phones
  • Mobile devices
  • Audio/video media
  • Voice mail
  • Computers
  • Software
  • Internet
  • Filing and records
  • Data storage/backup
  • Health
  • Public health system
  • Doctors
  • Dentist
  • Optometrist
  • Healthcare specialists
  • Checkups
  • Diet
  • Food
  • Exercise
  • Leisure
  • Books
  • Music
  • Video
  • Movies
  • TV shows
  • Hiking routes
  • Travel
  • Places to visit
  • People to visit
  • Web browsing
  • Photography
  • Sports equipment
  • Hobbies
  • Cooking
  • Recreation
  • Errands
  • Shopping
  • Stores
  • Hardware
  • Supplies
  • Groceries
  • Gifts
  • Pharmacy
  • Bank
  • Cleaners
  • Repairs
  • Financial
  • Worries
  • Taxes
  • Bills
  • Banks
  • Loans
  • Budget
  • Insurance
  • Mortgage
  • Bookkeeping
  • Investments
  • Accountants
  • Pets
  • Health
  • Supplies
  • Training
  • Legal
  • Wills
  • Trusts
  • Estate
  • Legal affairs
  • Friend/Family projects/activities
  • Friends
  • Partner
  • Parents
  • Relatives
  • Children
  • Home/household
  • Worries
  • Rent
  • Real estate
  • Repairs
  • Construction
  • Remodeling
  • Landlords
  • Heating and air conditioning
  • Plumbing
  • Utilities
  • Roof
  • Landscaping
  • Driveway
  • Garage
  • Walls
  • Floors
  • Ceilings
  • Decor
  • Furniture
  • Appliances
  • Lights and wiring
  • Kitchen supplies/equipment
  • Laundry
  • Purging, organizing, cleaning
  • Storage
  • Service providers
  • Personal development
  • Classes
  • Seminars
  • Education
  • Coaching/counseling
  • Career
  • Creative expressions
  • Transportation
  • Bicycles
  • Maintenance
  • Repair
  • Commuting
  • Motor vehicles
  • Clothes
  • Casual
  • Formal
  • Sports
  • Accessories
  • Luggage
  • Repairs
  • Tailoring
  • Professional
  • Community
  • Activism
  • Neighborhood
  • Neighbors
  • Voting
  • Waiting for
  • Product orders
  • Repairs
  • Reimbursements
  • Loaned items
  • Information
  • Projects/tasks completed by family/friends
Professional⚑
  • Projects started, not completed
  • Projects that need to be started
  • “Look into . . .” projects
  • Commitments/promises to others
  • Colleagues
  • Boss/partners
  • Others in organization
  • “Outside” people
    • Customers
    • Other organizations
    • Professionals
    • Vendors
  • Communications to make/get
  • Internal/external
  • Initiate or respond to:
    • Phone calls
    • Voice notes
    • E-mails
    • Text messages
    • Letters
    • Social media postings
  • Other writing to finish/submit
  • Reports
  • Evaluations/reviews
  • Proposals
  • Articles
  • Manuals/instructions
  • Summaries
  • Rewrites and edits
  • Status reporting
  • Conversation and communication tracking
  • Meetings that need to be set/requested
  • Who needs to know about what decisions?
  • Significant read/review
  • Professional development
  • Training/seminars
  • Things to learn
  • Things to find out
  • Skills to practice/develop
  • Books to read/study
  • Research
  • Formal education (licensing, degrees)
  • Career research
  • Résumé
  • Performance objectives
  • Financial
  • Forecasts/projections
  • Credit line
  • Stability
  • Planning/organizing
  • Formal planning (goals, targets, objectives)
  • Current projects (next stages)
  • Organizational initiatives
  • Upcoming events
  • Meetings
  • Presentations
  • Conferences
  • Organizational structuring
  • Changes in facilities
  • Installation of new systems/equipment
  • Travel
  • Vacation
  • Business trips
  • Waiting for . . .
  • Information
  • Delegated tasks/projects
  • Completions critical to projects
  • Answers to questions
  • Replies to:
    • E-mails
    • Letters
    • Proposals
    • Calls
    • Invitations
  • Requisitions
  • Reimbursements
  • Insurance claims
  • Ordered items
  • Repairs
  • Tickets
  • Decisions of others
  • Organization Development
  • Organization chart
  • Restructuring
  • Roles
  • Job descriptions
  • Facilities
  • New systems
  • Leadership
  • Change initiatives
  • Succession planning
  • Organization culture
  • Administration
  • Legal
  • Insurance
  • Personnel
  • Staffing
  • Policies/procedures
  • Training
  • Staff
  • Hiring/firing/promoting
  • Reviews
  • Communication
  • Staff development
  • Compensation
  • Feedback
  • Morale
  • Systems
  • Mobile devices
  • Phones
  • Computers
  • Software
  • Databases
  • Telecommunications
  • Internet
  • Filing and reference
  • Inventories
  • Storage
  • Office/site
  • Space/arrangements
  • Furniture
  • Equipment
  • Decorations
  • Utilities
  • Supplies
  • Maintenance/cleaning
  • Security

Now that you know where do your inputs come from you need to think how do you want to manage them from now on to ensure that you're able to be able to continuously capture items in a frictionless way.

Empty the inbox⚑

Now that we have collected everything that has your attention, you need to get to the bottom of your inbox. To be able to do it in a reasonable amount of time you are not meant to actually do the items themselves, instead analyze each item and decide what it is, what it means and what are you going to do with it.

Follow the steps of Clarify and organize until you've reached the bottom of your inbox. You’ll dump a mess of things, file a bunch, do a lot of two-minute actions, and hand off a number of items to other people. You’ll also wind up with a stack of items that have actions associated with them that you still need to do soon, someday, or on a specific date, and reminders of things you’re waiting on from other people. Now that you have everything at the next action level, we need to scale up in the abstraction ladder so that we can prioritize better what to do next.

Unclassified thoughts⚑

  • There must be zero resistance to using the systems we have. Having to continually reinvent our in-tray, our filing system, and how and where we process our stuff can only be a source of incessant distraction.
  • One of the best tricks for enhancing your productivity is having organizing tools you love to use.
  • Being organized means nothing more or less than where something is matches what it means to you.
  • Your organisation system is not something that you'll create all at once. It will evolve as you process yuor stuff and test out whether you have put everything in the best place for you. It won't remain static, it will evolve as you do.

The weekly review⚑

The Weekly Review is the time to:

  • Gather and process all your stuff.
  • Review your system.
  • Update your lists.
  • Get clean, clear, current, and complete.
Last update: 2023-11-27
\ No newline at end of file diff --git a/linux/zfs/index.html b/linux/zfs/index.html index 16eaa6f79c2..f7d2ff92c3f 100644 --- a/linux/zfs/index.html +++ b/linux/zfs/index.html @@ -1,4 +1,4 @@ - ZFS - The Blue Book
Skip to content

OpenZFS

OpenZFS is a file system with volume management capabilities designed specifically for storage servers.

Some neat features of ZFS include:

  • Aggregating multiple physical disks into a single filesystem.
  • Automatically repairing data corruption.
  • Creating point-in-time snapshots of data on disk.
  • Optionally encrypting or compressing data on disk.

Usage⚑

Mount a pool as readonly⚑

zpool import -o readonly=on {{ pool_name }}
+ ZFS - The Blue Book       
  
 
  
 
 
 
 
 
 
  
 
 
 
 
 
  
 
 
 
 
 
OpenZFS
 
OpenZFS is a file system with volume management capabilities designed specifically for storage servers.
 
Some neat features of ZFS include:
 
     
  • Aggregating multiple physical disks into a single filesystem.
    •  
  • Automatically repairing data corruption.
    •  
  • Creating point-in-time snapshots of data on disk.
    •  
  • Optionally encrypting or compressing data on disk.
    •  
 
Usage⚑
 
Mount a pool as readonly⚑
 
zpool import -o readonly=on {{ pool_name }}
 
 
Mount a ZFS snapshot in a directory as readonly⚑
 
mount -t zfs {{ pool_name }}/{{ snapshot_name }} {{ mount_path }} -o ro
 
 
Mount a dataset that is encrypted⚑
 
If your dataset is encrypted using a key file you need to:
 
     
  • Mount the device that has your keys
    •  
  • Import the pool without loading the key because you want to override the keylocation attribute with zfs load-key. Without the -l option, any encrypted datasets won't be mounted, which is what you want.
    •  
  • Load the key(s) for the dataset(s)
    •  
  • Mount the dataset(s).
    •  
 
zpool import rpool    # without the `-l` option!
 zfs load-key -L file:///path/to/keyfile rpool
@@ -177,7 +177,8 @@
 
 
If you need to share to multiple subnets, you would do something like:
 
sudo zfs set sharenfs="rw=@192.168.0.0/24,rw=@10.0.0.0/24" pool-name/dataset-name
 
 
If you need root to be able to write to the directory enable the no_root_squash NFS option
 
 
root_squash — Prevents root users connected remotely from having root privileges and assigns them the user ID for the user nfsnobody. This effectively "squashes" the power of the remote root user to the lowest local user, preventing unauthorized alteration of files on the remote server. Alternatively, the no_root_squash option turns off root squashing. To squash every remote user, including root, use the all_squash option. To specify the user and group IDs to use with remote users from a particular host, use the anonuid and anongid options, respectively. In this case, a special user account can be created for remote NFS users to share and specify (anonuid=,anongid=), where is the user ID number and is the group ID number.
 
 
You should now be able to mount the NFS export from an NFS client. Install the client with:
 
sudo apt-get install nfs-common
 
 
And then mount it with:
 
mount -t nfs hostname.example.com:/srv /mnt
-
 
To permanently mount it you need to add it to your /etc/fstab, check this section for more details.
 
Backup⚑
 
Please remember that RAID is not a backup, it guards against one kind of hardware failure. There's lots of failure modes that it doesn't guard against though:
 
     
  • File corruption
    •  
  • Human error (deleting files by mistake)
    •  
  • Catastrophic damage (someone dumps water onto the server)
    •  
  • Viruses and other malware
    •  
  • Software bugs that wipe out data
    •  
  • Hardware problems that wipe out data or cause hardware damage (controller malfunctions, firmware bugs, voltage spikes, ...)
    •  
 
That's why you still need to make backups.
 
ZFS has the builtin feature to make snapshots of the pool. A snapshot is a first class read-only filesystem. It is a mirrored copy of the state of the filesystem at the time you took the snapshot. They are persistent across reboots, and they don't require any additional backing store; they use the same storage pool as the rest of your data. 
 
If you remember ZFS's awesome nature of copy-on-write filesystems, you will remember the discussion about Merkle trees. A ZFS snapshot is a copy of the Merkle tree in that state, except we make sure that the snapshot of that Merkle tree is never modified.
 
Creating snapshots is near instantaneous, and they are cheap. However, once the data begins to change, the snapshot will begin storing data. If you have multiple snapshots, then multiple deltas will be tracked across all the snapshots. However, depending on your needs, snapshots can still be exceptionally cheap.
 
ZFS snapshot lifecycle management⚑
 
ZFS doesn't though have a clean way to manage the lifecycle of those snapshots. There are many tools to fill the gap:
 
     
  • sanoid: Made in Perl, 2.4k stars, last commit April 2022, last release April 2021
    •  
  • zfs-auto-snapshot: Made in Bash, 767 stars, last commit/release on September 2019
    •  
  • pyznap: Made in Python, 176 stars, last commit/release on September 2020
    •  
  • Custom scripts.
    •  
 
It seems that the state of the art of ZFS backups is not changing too much in the last years, possibly because the functionality is covered so there is no need for further development. So I'm going to manage the backups with sanoid despite it being done in Perl because it's the most popular, it looks simple but flexible for complex cases, and it doesn't look I'd need to tweak the code.
 
Restore a backup⚑
 
You can list the available snapshots of a filesystem with zfs list -t snapshot {{ pool_or_filesystem_name }}, if you don't specify the pool_or_filesystem_name it will show all available snapshots.
 
You have two ways to restore a backup:
  
mount -t zfs main/lyz@autosnap_2023-02-17_13:15:06_hourly /mnt
+
 
To permanently mount it you need to add it to your /etc/fstab, check this section for more details.
 
Backup⚑
 
Please remember that RAID is not a backup, it guards against one kind of hardware failure. There's lots of failure modes that it doesn't guard against though:
 
     
  • File corruption
    •  
  • Human error (deleting files by mistake)
    •  
  • Catastrophic damage (someone dumps water onto the server)
    •  
  • Viruses and other malware
    •  
  • Software bugs that wipe out data
    •  
  • Hardware problems that wipe out data or cause hardware damage (controller malfunctions, firmware bugs, voltage spikes, ...)
    •  
 
That's why you still need to make backups.
 
ZFS has the builtin feature to make snapshots of the pool. A snapshot is a first class read-only filesystem. It is a mirrored copy of the state of the filesystem at the time you took the snapshot. They are persistent across reboots, and they don't require any additional backing store; they use the same storage pool as the rest of your data. 
 
If you remember ZFS's awesome nature of copy-on-write filesystems, you will remember the discussion about Merkle trees. A ZFS snapshot is a copy of the Merkle tree in that state, except we make sure that the snapshot of that Merkle tree is never modified.
 
Creating snapshots is near instantaneous, and they are cheap. However, once the data begins to change, the snapshot will begin storing data. If you have multiple snapshots, then multiple deltas will be tracked across all the snapshots. However, depending on your needs, snapshots can still be exceptionally cheap.
 
ZFS snapshot lifecycle management⚑
 
ZFS doesn't though have a clean way to manage the lifecycle of those snapshots. There are many tools to fill the gap:
 
     
  • sanoid: Made in Perl, 2.4k stars, last commit April 2022, last release April 2021
    •  
  • zfs-auto-snapshot: Made in Bash, 767 stars, last commit/release on September 2019
    •  
  • pyznap: Made in Python, 176 stars, last commit/release on September 2020
    •  
  • Custom scripts.
    •  
 
It seems that the state of the art of ZFS backups is not changing too much in the last years, possibly because the functionality is covered so there is no need for further development. So I'm going to manage the backups with sanoid despite it being done in Perl because it's the most popular, it looks simple but flexible for complex cases, and it doesn't look I'd need to tweak the code.
 
Remove all snapshots of a dataset⚑
 
zfs list -t snapshot -o name path/to/dataset | tail -n+2 | tac | xargs -n 1 zfs destroy -r
+
 
Restore a backup⚑
 
You can list the available snapshots of a filesystem with zfs list -t snapshot {{ pool_or_filesystem_name }}, if you don't specify the pool_or_filesystem_name it will show all available snapshots.
 
You have two ways to restore a backup:
  
mount -t zfs main/lyz@autosnap_2023-02-17_13:15:06_hourly /mnt
 
 
To umount the snapshot run umount /mnt.
 
     
  • Rolling back the filesystem to the snapshot state: Rolling back to a previous snapshot will discard any data changes between that snapshot and the current time. Further, by default, you can only rollback to the most recent snapshot. In order to rollback to an earlier snapshot, you must destroy all snapshots between the current time and that snapshot you wish to rollback to. If that's not enough, the filesystem must be unmounted before the rollback can begin. This means downtime.
    •  
 
To rollback the "tank/test" dataset to the "tuesday" snapshot, we would issue:
 
$: zfs rollback tank/test@tuesday
 cannot rollback to 'tank/test@tuesday': more recent snapshots exist
 use '-r' to force deletion of the following snapshots:
@@ -208,4 +209,4 @@
 sudo zpool export WD_1TB
 sudo rm disk2s1
 sudo zpool import WD_1TB
-
 
If you don't care about the zpool anymore, sadly your only solution is to reboot the server. Real ugly, so be careful when you umount zpools.
 
Learning⚑
 
I've found that learning about ZFS was an interesting, intense and time consuming task. If you want a quick overview check this video. If you prefer to read, head to the awesome Aaron Toponce articles and read all of them sequentially, each is a jewel. The docs on the other hand are not that pleasant to read. For further information check JRS articles.
 
Resources⚑
  
 
  Last update: 2023-10-19  
 
 
 
  
  
 
 
 
      
\ No newline at end of file
+

If you don't care about the zpool anymore, sadly your only solution is to reboot the server. Real ugly, so be careful when you umount zpools.

Learning⚑

I've found that learning about ZFS was an interesting, intense and time consuming task. If you want a quick overview check this video. If you prefer to read, head to the awesome Aaron Toponce articles and read all of them sequentially, each is a jewel. The docs on the other hand are not that pleasant to read. For further information check JRS articles.

Resources⚑

Last update: 2023-11-27
\ No newline at end of file diff --git a/orgmode/index.html b/orgmode/index.html index 91b72e8107f..813ac217db7 100644 --- a/orgmode/index.html +++ b/orgmode/index.html @@ -1,4 +1,4 @@ - Org Mode - The Blue Book
Skip to content

Org Mode

nvim-orgmode is a Orgmode clone written in Lua for Neovim. Org-mode is a flexible note-taking system that was originally created for Emacs. It has gained wide-spread acclaim and was eventually ported to Neovim. This page is heavily focused to the nvim plugin, but you can follow the concepts for emacs as well.

If you use Android try orgzly.

Installation⚑

Add to your plugin config:

use {'nvim-orgmode/orgmode', config = function()
+ Org Mode - The Blue Book       
  
 
  
 
 
 
 
 
 
  
 
 
 
 
 
  
 
 
 
 
 
Org Mode
 
nvim-orgmode is a Orgmode clone written in Lua for Neovim. Org-mode is a flexible note-taking system that was originally created for Emacs. It has gained wide-spread acclaim and was eventually ported to Neovim. This page is heavily focused to the nvim plugin, but you can follow the concepts for emacs as well.
 
If you use Android try orgzly.
 
Installation⚑
 
Add to your plugin config:
 
use {'nvim-orgmode/orgmode', config = function()
   require('orgmode').setup{}
 end
 }
@@ -176,11 +176,11 @@
 
 
You can't yet manage the checkboxes as you do the headings by promoting, demoting and moving them around.
 
Follow this issue if you want to see the progress of it's children at the parent checkbox.
  
One final aspect of the org file syntax are links. Links are of the form [[link][description]], where link can be an:
  
A link that does not look like a URL refers to the current document. You can follow it with gx when point is on the link (Default <leader>oo) if you use the next configuration.
 
org = {
   org_open_at_point = 'gx',
 }
-
  
Org provides several refinements to internal navigation within a document. Most notably:
 
     
  • [[*Some section]]: points to a headline with the name Some section.
    •  
  • [[#my-custom-id]]: targets the entry with the CUSTOM_ID property set to my-custom-id. 
    •  
 
When the link does not belong to any of the cases above, Org looks for a dedicated target: the same string in double angular brackets, like <<My Target>>.
 
If no dedicated target exists, the link tries to match the exact name of an element within the buffer. Naming is done, unsurprisingly, with the NAME keyword, which has to be put in the line before the element it refers to, as in the following example
 
#+NAME: My Target
+
  
Org provides several refinements to internal navigation within a document. Most notably:
 
     
  • [[Some section]]: points to a headline with the name Some section.
    •  
  • [[#my-custom-id]]: targets the entry with the CUSTOM_ID property set to my-custom-id. 
    •  
 
When the link does not belong to any of the cases above, Org looks for a dedicated target: the same string in double angular brackets, like <<My Target>>.
 
If no dedicated target exists, the link tries to match the exact name of an element within the buffer. Naming is done, unsurprisingly, with the NAME keyword, which has to be put in the line before the element it refers to, as in the following example
 
#+NAME: My Target
 | a  | table      |
 |----+------------|
 | of | four cells |
-
 
Ultimately, if none of the above succeeds, Org searches for a headline that is exactly the link text but may also include a TODO keyword and tags, or initiates a plain text search.
 
Note that you must make sure custom IDs, dedicated targets, and names are unique throughout the document. Org provides a linter to assist you in the process, if needed, but I have not searched yet one for nvim.
  
     
  • URL (http://, https://)
    •  
  • Path to a file (file:/path/to/org/file). File links can contain additional information to jump to a particular location in the file when following a link. This can be:
    •  
  • file:~/code/main.c::255: A line number 
    •  
  • file:~/xx.org::My Target: A search for <<My Target>>
    •  
  • file:~/xx.org::#my-custom-id: A search for- a custom ID
    •  
 
Properties⚑
 
Properties are key-value pairs associated with an entry. They live in a special drawer with the name PROPERTIES. Each property is specified on a single line, with the key (surrounded by colons) first, and the value after it:
 
* CD collection
+
 
Ultimately, if none of the above succeeds, Org searches for a headline that is exactly the link text but may also include a TODO keyword and tags, or initiates a plain text search.
 
Note that you must make sure custom IDs, dedicated targets, and names are unique throughout the document. Org provides a linter to assist you in the process, if needed, but I have not searched yet one for nvim.
  
     
  • URL (http://, https://)
    •  
  • Path to a file (file:/path/to/org/file). File links can contain additional information to jump to a particular location in the file when following a link. This can be:
    •  
  • file:~/code/main.c::255: A line number 
    •  
  • file:~/xx.org::*My Target: A search for <<My Target>> heading.
    •  
  • file:~/xx.org::#my-custom-id: A search for- a custom ID
    •  
 
Properties⚑
 
Properties are key-value pairs associated with an entry. They live in a special drawer with the name PROPERTIES. Each property is specified on a single line, with the key (surrounded by colons) first, and the value after it:
 
* CD collection
 ** Classic
 *** Goldberg Variations
     :PROPERTIES:
@@ -295,7 +295,7 @@
     target = "~/org/repos.org",
   }
 }
-
 
Use capture⚑
 
Synchronize with external calendars⚑
 
You may want to synchronize your calendar entries with external ones shared with other people, such as nextcloud calendar or google.
 
The orgmode docs have a tutorial to sync with google and suggests some orgmode packages that do that, sadly it won't work with nvim-orgmode. We'll need to go the "ugly way" by:
  
Importing the ics to orgmode⚑
 
There are many tools that do this:
  
They import an ics file
 
Exporting from orgmode to ics⚑
 
Other interesting features⚑
 
Some interesting features for the future are:
  
Troubleshooting⚑
 
Create an issue in the orgmode repository⚑
 
     
  • Create a new issue
    •  
  • Create the minimal_init.lua file from this file 
    vim.cmd([[set runtimepath=$VIMRUNTIME]])
+
     
    Use capture⚑
     
    The orgmode repository file organization⚑
     
    How to structure the different orgmode files is something that has always confused me, each one does it's own way, and there are no good posts on why one structure is better than other, people just state what they do.
     
    I've started with a typical gtd structure with a directory for the todo another for the calendar then another for the references. In the todo I had a file for personal stuff, another for each of my work clients, and the someday.org. Soon making the internal links was cumbersome so I decided to merge the personal todo.org and the someday.org into the same file and use folds to hide uninteresting parts of the file. The reality is that I feel that orgmode is less responsive and that I often feel lost in the file. 
     
    I'm now more into the idea of having files per project in a flat structure and use an index.org file to give it some sense in the same way I do with the mkdocs repositories. Then I'd use internal links in the todo.org file to organize the priorities of what to do next.
     
    Benefits:
     
       
    • As we're using a flat structure at file level, the links between the files are less cumbersome file:./project.org::*heading. We only need to have unique easy to remember names for the files, instead of having to think on which directory was the file I want to make the link to. The all in one file structure makes links even easier, just *heading, but the disadvantages make it not worth it.
      •  
    • You have the liberty to have a generic link like Work on project or if you want to fine grain it, link the specific task of the project
      •  
    • The todo file will get smaller.
      •  
    • It has been the natural evolution of other knowledge repositories such as blue
      •  
     
    Cons:
     
       
    • Filenames must be unique. It hasn't been a problem in blue.
      •  
    • Blue won't be flattened into Vida as it's it's own knowledge repository
      •  
     
    Synchronizations⚑
     
    Synchronize with other orgmode repositories⚑
     
    I use orgmode both at the laptop and the mobile, I want to syncronize some files between both with the next requisites:
     
       
    • The files should be available on the devices when I'm not at home
      •  
    • The synchronization will be done only on the local network
      •  
    • The synchronization mechanism will only be able to see the files that need to be synched. 
      •  
    • Different files can be synced to different devices. If I have three devices (laptop, mobile, tablet) I want to sync all mobile files to the laptop but just some to the tablet).
      •  
     
    Right now I'm already using syncthing to sync files between the mobile and my server, so it's tempting to use it also to solve this issue. So the first approach is to spawn a syncthing docker at the laptop that connects with the server to sync the files whenever I'm at home. 
     
    Mount the whole orgmode repository with syncthing⚑
     
    I could mount the whole orgmode directory and use the ignore patterns of syncthing, but that will make syncthing see more files than I want even though it won't sync them to the other devices. The ideal scenario is where syncthing only sees the files that needs to sync, so that in case of a vulnerability only a subset of the files is leaked.
     
    Mount a specific directory to sync⚑
     
    An alternative would be to have a mobile directory at the orgmode repository where the files that need to be synced will live. The problem is that it would break the logical structure of the repository and it would make difficult to make internal links between files as you need to remember or need to search if the file is in the usual place or in the mobile directory. To avoid this we could use hard links. Soft links don't work well because:
     
       
    • If you have the file in the org repo and do the soft link in the mobile directory, syncthing won't know what to do with it
      •  
    • If you have the file in the mobile repo and do the soft link in the repository, nvim-orgmode won't be able to work well with the file. I don't know why but those files don't show when I search them in telescope (and I have symbolic links enabled in the config).
      •  
     
    If we use this workflow, we'd need to manually create the hard links each time a new file is created that needs to be linked
     
    This is also a good solution for the different authorization syncs as you can only have one syncthing directory per Linux directory so if you want different authorization for different devices you won't be able to do this unless you create a specific directory for that share. For example if I want to have only one file shared to the tablet I'd need a tablet directory.
     
    Select which files to mount on the docker command⚑
     
    We could also select which files to mount on the syncthing docker of the laptop. I find this to be an ugly solution because we'd first need to mount a directory so that syncthing can write it's internal data and then map each of the files we want to sync. So each time a new file is added, we need to change the docker command... Unpleasant.
     
    Use the org-orgzly script⚑
     
    Another solution would be to use org-orgzly script to parse a chosen org file or files, check if an entry meets required parameters, and if it does, write the entry in a new file located inside the directory you desire to sync with orgzly. In theory it may work but I feel it's too Dropbox focused.
     
    Synchronize with external calendars⚑
     
    You may want to synchronize your calendar entries with external ones shared with other people, such as nextcloud calendar or google.
     
    The orgmode docs have a tutorial to sync with google and suggests some orgmode packages that do that, sadly it won't work with nvim-orgmode. We'll need to go the "ugly way" by:
      
    Importing the ics to orgmode⚑
     
    There are many tools that do this:
      
    They import an ics file
     
    Exporting from orgmode to ics⚑
     
    Other interesting features⚑
     
    Some interesting features for the future are:
      
    Troubleshooting⚑
     
    Create an issue in the orgmode repository⚑
     
       
    • Create a new issue
      •  
    • Create the minimal_init.lua file from this file 
      vim.cmd([[set runtimepath=$VIMRUNTIME]])
 vim.cmd([[set packpath=/tmp/nvim/site]])
 
 local package_root = '/tmp/nvim/site/pack'
@@ -355,4 +355,4 @@
   load_plugins()
   load_config()
 end
-
      •  
    • Add the leader configuration at the top of the file vim.g.mapleader = ' '
      •  
    • Open it with nvim -u minimal_init.lua
      •  
     
    Sometimes  doesn't work⚑
     
    Close the terminal and open a new one (pooooltergeist!).
     
    Narrow/Widen to subtree⚑
     
    It's not yet supported to focus or zoom on one task.
     
    Comparison with Markdown⚑
     
    What I like of Org mode over Markdown:
     
       
    • The whole interface to interact with the elements of the document through key bindings:
      •  
    • Move elements around.
      •  
    • Create elements
      •  
    • The TODO system is awesome
      •  
    • The Agenda system
      •  
    • How it handles checkboxes <3
      •  
    • Easy navigation between references in the document
      •  
    • Archiving feature
      •  
    • Refiling feature
      •  
    • # is used for comments.
      •  
    • Create internal document links is easier, you can just copy and paste the heading similar to [[*This is the heading]] on markdown you need to edit it to [](#this-is-the-heading).
      •  
     
    What I like of markdown over Org mode:
     
       
    • The syntax of the headings ## Title better than ** Title. Although it makes sense to have # for comments.
      •  
    • The syntax of the links: [reference](link) is prettier to read and write than [[link][reference]], although this can be improved if only the reference is shown by your editor (nvim-orgmode doesn't do his yet)
      •  
     
    Interesting things to investigate⚑
     
       
    • org-bullets.nvim: Show org mode bullets as UTF-8 characters.
      •  
    • headlines.nvim: Add few highlight options for code blocks and headlines.
      •  
    • Sniprun: A neovim plugin to run lines/blocs of code (independently of the rest of the file), supporting multiples languages.
      •  
     
    References⚑
      
     
      Last update: 2023-10-19  
     
 
 
  
  
 
 
 
      
\ No newline at end of file
+
  • Add the leader configuration at the top of the file vim.g.mapleader = ' '
  • Open it with nvim -u minimal_init.lua

    • Sometimes doesn't work⚑

    Close the terminal and open a new one (pooooltergeist!).

    Narrow/Widen to subtree⚑

    It's not yet supported to focus or zoom on one task.

    Comparison with Markdown⚑

    What I like of Org mode over Markdown:

    • The whole interface to interact with the elements of the document through key bindings:
    • Move elements around.
    • Create elements
    • The TODO system is awesome
    • The Agenda system
    • How it handles checkboxes <3
    • Easy navigation between references in the document
    • Archiving feature
    • Refiling feature
    • # is used for comments.
    • Create internal document links is easier, you can just copy and paste the heading similar to [[*This is the heading]] on markdown you need to edit it to [](#this-is-the-heading).

    What I like of markdown over Org mode:

    • The syntax of the headings ## Title better than ** Title. Although it makes sense to have # for comments.
    • The syntax of the links: [reference](link) is prettier to read and write than [[link][reference]], although this can be improved if only the reference is shown by your editor (nvim-orgmode doesn't do his yet)

    Interesting things to investigate⚑

    • org-bullets.nvim: Show org mode bullets as UTF-8 characters.
    • headlines.nvim: Add few highlight options for code blocks and headlines.
    • Sniprun: A neovim plugin to run lines/blocs of code (independently of the rest of the file), supporting multiples languages.

    References⚑

    Last update: 2023-11-27
    \ No newline at end of file diff --git a/rtorrent/index.html b/rtorrent/index.html index 2cf49cd4bd1..b20c1803554 100644 --- a/rtorrent/index.html +++ b/rtorrent/index.html @@ -1,2 +1,2 @@ - Rtorrent - The Blue Book
    Skip to content

    Rtorrent

    Debugging⚑

    • Get into the docker with docker exec -it docker_name bash
    • cd /home/nobody

    • Open the rtorrent.sh file add set -x above the line you think is starting your rtorrent and set +x below to fetch the command that is launching your rtorrent instance, for example:

      /usr/bin/tmux new-session -d -s rt -n rtorrent /usr/bin/rtorrent -b 12.5.232.12 -o ip=232.234.324.211
-

    If you manually run /usr/bin/rtorrent -b 12.5.232.12 -o ip=232.234.324.211 you'll get more information on why rtorrent is not starting.

    Last update: 2022-03-04
    \ No newline at end of file + Rtorrent - The Blue Book
    Skip to content

    Rtorrent

    Debugging⚑

    • Get into the docker with docker exec -it docker_name bash
    • cd /home/nobody

    • Open the rtorrent.sh file add set -x above the line you think is starting your rtorrent and set +x below to fetch the command that is launching your rtorrent instance, for example:

      /usr/bin/tmux new-session -d -s rt -n rtorrent /usr/bin/rtorrent -b 12.5.232.12 -o ip=232.234.324.211
+

    If you manually run /usr/bin/rtorrent -b 12.5.232.12 -o ip=232.234.324.211 you'll get more information on why rtorrent is not starting.

    References⚑

    Last update: 2023-11-27
    \ No newline at end of file diff --git a/search/search_index.json b/search/search_index.json index 821ee3d1a7d..6696c708fd7 100644 --- a/search/search_index.json +++ b/search/search_index.json @@ -1 +1 @@ -{"config": {"lang": ["en"], "separator": "[\\s\\-]+", "pipeline": ["stopWordFilter"]}, "docs": [{"location": "", "title": "Introduction", "text": "

    This is my personal digital garden where I share everything I learn about a huge variety of topics, from antifascism and other forms of activism, to life management, health, Python, DevOps, software architecture, writing, dancing, or data analysis.

    Unlike in common blogging where you write an article and forget about it, posts are treated as plants in various stages of growth and nurturing. Some might wither and die, and others will flourish and provide a source of continued knowledge for the gardener and folks in the community that visit.

    To follow the updates of this site, subscribe to any of the RSS feeds.

    "}, {"location": "#visiting-the-garden", "title": "Visiting the garden", "text": "

    If this is your first visit, welcome!, you may be overwhelmed by the amount of content and don't know where to start reading. Start with the first article that grabs your attention of the navigation tree on the left and be ready to follow the internal links to read the rest.

    Or you can use it as a reference, by using the top search field or by cloning the git repository and using grep like tools.

    "}, {"location": "#make-your-own-digital-garden", "title": "Make your own digital garden", "text": "

    Don't be afraid to create one of your own and share what you know with the world. Or contribute to existing ones. I would love to see the blue-book maintained by other people too.

    If you don't want to start from scratch, you can fork the blue book, change the name and start writing straight away.

    You can view other similar gardens to get inspiration.

    "}, {"location": "#history", "title": "History", "text": "

    I've tried writing blogs in the past, but it doesn't work for me. I can't stand the idea of having to save some time to sit and write a full article of something I've done. I need to document at the same time as I develop or learn. As I usually use incremental reading or work on several projects, I don't write one article, but improve several at the same time in a unordered way. That's why I embrace Gwern's Long Content principle.

    The only drawback of this format is that there is no friendly way for a reader to keep updated with the changes, that's why I created mkdocs-newsletter.

    In 2016 I started writing in text files summaries of different concepts, how to install or how to use tools. In the beginning it was plaintext, then came Markdown, then Asciidoc, I did several refactors to reorder the different articles based in different structured ways, but I always did it with myself as the only target audience.

    Three years, 7422 articles and almost 50 million lines later, I found Gwern's website and Nikita's wiki, which made me think that it was time to do another refactor to give my wiki a semantical structure, go beyond a simple reference making it readable and open it to the internet.

    And the blue book was born.

    "}, {"location": "#contributing", "title": "Contributing", "text": "

    If you find a mistake or want to add new content, please make the changes. You can use the edit button on the top right of any article to add them in a pull request, if you don't know what that means, you can always open an issue or send me an email.

    "}, {"location": "#thank-you", "title": "Thank you", "text": "

    If you liked my book and want to show your support, please see if you know how can I fulfill any item of my wish list or contribute to my other projects.

    Another way to say \"thanks\" is to give me a donation.

    You can use

    or

    If you are using some of my open-source tools or documentation, have enjoyed them, and want to say \"thanks\", this is a very strong way to do it.

    If your product/company depends on these tools, you can sponsor me to ensure I keep happily maintaining them.

    If these tools are helping you save money, time, effort, or frustrations; or they are helping you make money, be more productive, efficient, secure, enjoy a bit more your work, or get your product ready faster, this is a great way to show your appreciation. Thanks for that!

    "}, {"location": "abstract_syntax_trees/", "title": "Abstract syntax trees", "text": "

    Abstract syntax trees (AST) is a tree representation of the abstract syntactic structure of text (often source code) written in a formal language. Each node of the tree denotes a construct occurring in the text.

    The syntax is \"abstract\" in the sense that it does not represent every detail appearing in the real syntax, but rather just the structural or content-related details. For instance, grouping parentheses are implicit in the tree structure, so these do not have to be represented as separate nodes. Likewise, a syntactic construct like an if-condition-then statement may be denoted by means of a single node with three branches.

    This distinguishes abstract syntax trees from concrete syntax trees, traditionally designated parse trees. Parse trees are typically built by a parser during the source code translation and compiling process. Once built, additional information is added to the AST by means of subsequent processing, e.g., contextual analysis.

    Abstract syntax trees are also used in program analysis and program transformation systems.

    "}, {"location": "abstract_syntax_trees/#how-to-construct-an-ast", "title": "How to construct an AST", "text": "

    TBD

    pyparsing looks to be a good candidate to implement ASTs.

    "}, {"location": "activitywatch/", "title": "ActivityWatch", "text": "

    ActivityWatch is a bundle of software that tracks your computer activity. You are, by default, the sole owner of your data.

    ActivityWatch is:

    "}, {"location": "activitywatch/#installation", "title": "Installation", "text": "

    It will start the web interface at http://localhost:5600 and will capture the data.

    "}, {"location": "activitywatch/#configuration", "title": "Configuration", "text": "

    First go to the settings page of the Web UI, you can define there the rules for the categories.

    More advanced settings can be changed on the files, but I had no need to go there yet.

    The used directories are:

    "}, {"location": "activitywatch/#watchers", "title": "Watchers", "text": "

    By default ActivityWatch comes with the next watchers:

    But you can add more, such as:

    They even show you how to create your own watcher.

    "}, {"location": "activitywatch/#syncing", "title": "Syncing", "text": "

    There is currently no syncing support. You'll need to export the data (under Raw Data, Export all buckets as JSON), and either tweak it so it can be imported, or analyze the data through other processes.

    "}, {"location": "activitywatch/#issues", "title": "Issues", "text": ""}, {"location": "activitywatch/#references", "title": "References", "text": ""}, {"location": "adr/", "title": "ADR", "text": "

    ADR are short text documents that captures an important architectural decision made along with its context and consequences.

    The whole document should be one or two pages long. Written as if it is a conversation with a future developer. This requires good writing style, with full sentences organized into paragraphs. Bullets are acceptable only for visual style, not as an excuse for writing sentence fragments.

    Pros:

    Cons:

    "}, {"location": "adr/#how-to-use-them", "title": "How to use them", "text": "

    We will keep a collection of architecturally significant decisions, those that affect the structure, non-functional characteristics, dependencies, interfaces or construction techniques.

    There are different templates you can start with, being the most popular Michael Nygard's one.

    The documents are stored in the project repository under the doc/arch directory, with a name convention of NNN-title_with_underscores.md, where NNN is a monotonically increasing number.

    If a decision is reversed, we'll keep the old one around, but mark it as superseded, as it's still relevant to know that it was a decision, but is no longer.

    "}, {"location": "adr/#adr-template", "title": "ADR template", "text": "

    Using Michael Nygard's template as a starting point, I'm going to use these sections:

    I'm using the following Ultisnip vim snippet:

    snippet adr \"ADR\"\nDate: `date +%Y-%m-%d`\n\n# Status\n<!-- What is the status? Draft, Proposed, Accepted, Rejected, Deprecated or Superseded?\n-->\n$1\n\n# Context\n<!-- What is the issue that we're seeing that is motivating this decision or change? -->\n$0\n\n# Proposals\n<!-- What are the possible solutions to the problem described in the context -->\n\n# Decision\n<!-- What is the change that we're proposing and/or doing? -->\n\n# Consequences\n<!-- What becomes easier or more difficult to do because of this change? -->\nendsnippet\n
    "}, {"location": "adr/#usage-in-a-project", "title": "Usage in a project", "text": "

    When starting a project, I'll do it by the ADRs, that way you evaluate the problem, structure the idea and leave a record of your initial train of thought.

    I found useful to:

    As the project starts to grow, the relationships between the ADRs will get more complex, it's useful to create an ADR landing page, where the user can follow the logic between them. MermaidJS can be used to create a nice diagram that shows this information.

    In the mkdocs-newsletter I've used the next structure:

    graph TD\n    001[001: High level analysis]\n    002[002: Initial MkDocs plugin design]\n    003[003: Selected changes to record]\n    004[004: Article newsletter structure]\n    005[005: Article newsletter creation]\n\n    001 -- Extended --> 002\n    002 -- Extended --> 003\n    002 -- Extended --> 004\n    002 -- Extended --> 005\n    003 -- Extended --> 004\n    004 -- Extended --> 005\n\n    click 001 \"https://lyz-code.github.io/mkdocs-newsletter/adr/001-initial_approach\" _blank\n    click 002 \"https://lyz-code.github.io/mkdocs-newsletter/adr/002-initial_plugin_design\" _blank\n    click 003 \"https://lyz-code.github.io/mkdocs-newsletter/adr/003-select_the_changes_to_record\" _blank\n    click 004 \"https://lyz-code.github.io/mkdocs-newsletter/adr/004-article_newsletter_structure\" _blank\n    click 005 \"https://lyz-code.github.io/mkdocs-newsletter/adr/005-create_the_newsletter_articles\" _blank\n\n    001:::accepted\n    002:::accepted\n    003:::accepted\n    004:::accepted\n    005:::accepted\n\n    classDef draft fill:#CDBFEA;\n    classDef proposed fill:#B1CCE8;\n    classDef accepted fill:#B1E8BA;\n    classDef rejected fill:#E8B1B1;\n    classDef deprecated fill:#E8B1B1;\n    classDef superseeded fill:#E8E5B1;\n

    Where we define:

    "}, {"location": "adr/#tools", "title": "Tools", "text": "

    Although adr-tools exist, I feel it's an overkill to create new documents and search on an existing codebase. We are now used to using other tools for the similar purpose, like Vim or grep.

    "}, {"location": "adr/#references", "title": "References", "text": ""}, {"location": "aerial_silk/", "title": "Aerial Silk", "text": "

    Aerial Silk is a type of performance in which one or more artists perform aerial acrobatics while hanging from a fabric. The fabric may be hung as two pieces, or a single piece, folded to make a loop, classified as hammock silks. Performers climb the suspended fabric without the use of safety lines and rely only on their training and skill to ensure safety. They use the fabric to wrap, suspend, drop, swing, and spiral their bodies into and out of various positions. Aerial silks may be used to fly through the air, striking poses and figures while flying. Some performers use dried or spray rosin on their hands and feet to increase the friction and grip on the fabric.

    "}, {"location": "aerial_silk/#warming-up", "title": "Warming up", "text": ""}, {"location": "aerial_silk/#arm-twist", "title": "Arm twist", "text": "

    . Leave the silk at your left, take it with the left hand with your arm straight up and you thumb pointing away from you. . Start twisting in z > 0 your arm from your shoulder until the thumb points to your front. . Keep on twisting and follow the movement with the rest of your upper body until you're hanging from that arm and stretching. You shouldn't move your feet in the whole process. . Repeat with the other arm.

    "}, {"location": "aerial_silk/#ball-controlled-inversions", "title": "Ball controlled inversions", "text": "

    . From standing position with each hand in a tissue, give it two or three loops to each hand and invert passing your legs between each tissue. . Bend your knees so that you become a ball and from that position. . While alive: . Keep on rotating 90 degrees more until your shins are parallel to the ground facing down. . Change the direction of the rotation and rotate 180 degrees until your shins are parallel to the ground but facing up.

    "}, {"location": "aerial_silk/#inverted-arm-twist-warmup", "title": "Inverted arm twist warmup", "text": "

    . From standing position with each hand in a tissue, give it two or three loops to each hand and invert passing your legs between each tissue. . Move your legs up until your body is parallel to the tissues and your shoulders are rotated back so your chest goes forward. . While alive: . Start lowering your feet using the tissues as a guide, while doing so start twisting your arms at shoulder level so that your chest goes to your back in a cat like position until your body limit. . Go back up doing the twist in the other direction till you're back up.

    "}, {"location": "aerial_silk/#inverted-knee-to-elbow", "title": "Inverted knee to elbow", "text": "

    . From standing position with each hand in a tissue, give it two or three loops to each hand and invert passing your legs between each tissue. . Move your legs up until your body is parallel to the tissues and your shoulders are rotated back so your chest goes forward. . While alive: . Start lowering your feet using the tissues as a guide for your knees until they are at elbow level. Don't bend your knees! . Go back up.

    "}, {"location": "aerial_silk/#horizontal-pull-ups", "title": "Horizontal pull-ups", "text": "

    . From standing position with each hand in a tissue, ask for a partner to take your legs until you're horizontal hanging from your hands and feet. . While alive: . Keeping your body straight, do a pull up with your arms. . Slowly unbend your elbows and stretch back your arms.

    The next level would be that you use your leg strength to grip your partner's hips instead of her holding your feet.

    "}, {"location": "aerial_silk/#basic-movements", "title": "Basic movements", "text": ""}, {"location": "aerial_silk/#splitting-the-tissues", "title": "Splitting the tissues", "text": "

    . Go to the desired height and get into a comfortable position, such as seated over your feet on a Russian climb. . Open the palm of one hand at eye level holding the whole tissue . Keep your bellybutton close to the tissue . With the other hand pinch the side of the silk and start walking with your fingers till you find the break between tissues.

    "}, {"location": "aerial_silk/#figures", "title": "Figures", "text": ""}, {"location": "aerial_silk/#waist-lock", "title": "Waist lock", "text": "

    . Climb to the desired height . Leave the tissue to the side you want to do the lock to, for example the left (try the other side as well). . Leave your right hand above your left, with your arm straight. Your left hand should be at breast level with your elbow bent. . With straight legs with pointed toes, bring your left leg a little bit to the front, while the right one catches the silk . Bright the right leg up trying to get the silk as close to your waist as possible. . Rotate your body in z>0 towards your left hand until you're looking down . Release your hands.

    "}, {"location": "aerial_silk/#ideas", "title": "Ideas", "text": ""}, {"location": "aerial_silk/#locking-shoulder-blades-when-gripping-the-silks", "title": "Locking shoulder blades when gripping the silks", "text": "

    When you are going to hang yourself from your hands:

    . Unlock your shoulder blades moving your chest to the back. . Embrace the silk with your arms twisting your hands inwards . Grip the silk and lock back your shoulder blades together as if you were holding an apple between them. That movement will make your hands twist in the other direction until your wrists are between you and the tissue.

    "}, {"location": "aerial_silk/#safety", "title": "Safety", "text": ""}, {"location": "afew/", "title": "afew", "text": "

    afew is an initial tagging script for notmuch mail.

    Its basic task is to provide automatic tagging each time new mail is registered with notmuch. In a classic setup, you might call it after notmuch new in an offlineimap post sync hook.

    It can do basic thing such as adding tags based on email headers or maildir folders, handling killed threads and spam.

    In move mode, afew will move mails between maildir folders according to configurable rules that can contain arbitrary notmuch queries to match against any searchable attributes.

    "}, {"location": "afew/#installation", "title": "Installation", "text": "

    First install the requirements:

    sudo apt-get install notmuch python3-notmuch\n

    Then configure notmuch.

    Finally install the program:

    pip3 install afew\n
    "}, {"location": "afew/#usage", "title": "Usage", "text": "

    To tag new emails use:

    afew -v --tag --new\n
    "}, {"location": "afew/#references", "title": "References", "text": ""}, {"location": "age_of_empires/", "title": "Age of Empires", "text": ""}, {"location": "age_of_empires/#basic-principles", "title": "Basic principles", "text": ""}, {"location": "age_of_empires/#openings-or-build-orders", "title": "Openings or Build Orders", "text": ""}, {"location": "age_of_empires/#basic-opening", "title": "Basic Opening", "text": "

    When the match starts:

    Each additional worker constructing a building will help in 100/(n + 1) %, so the second worker will make the building come up a 25% faster.

    "}, {"location": "age_of_empires/#fast-castle-boom", "title": "Fast castle boom", "text": ""}, {"location": "age_of_empires/#strategy-guides", "title": "Strategy guides", "text": ""}, {"location": "age_of_empires/#how-to-play-maps", "title": "How to play maps", "text": ""}, {"location": "age_of_empires/#inside-the-mind-of-a-pro-player", "title": "Inside the mind of a pro player", "text": ""}, {"location": "age_of_empires/#strategies-against-civilisations", "title": "Strategies against civilisations", "text": "

    I'm using only the mongols, and so far I've seen/heard from the pros the next strategies:

    "}, {"location": "age_of_empires/#newbie-pitfalls", "title": "Newbie pitfalls", "text": ""}, {"location": "age_of_empires/#micromanagements", "title": "Micromanagements", "text": ""}, {"location": "age_of_empires/#workers", "title": "Workers", "text": ""}, {"location": "age_of_empires/#sheep-hunting", "title": "Sheep hunting", "text": ""}, {"location": "age_of_empires/#lumber-jacking", "title": "Lumber jacking", "text": ""}, {"location": "age_of_empires/#berry-gathering", "title": "Berry gathering", "text": ""}, {"location": "age_of_empires/#boar-hunting", "title": "Boar hunting", "text": ""}, {"location": "age_of_empires/#deer-hunting", "title": "Deer hunting", "text": ""}, {"location": "age_of_empires/#house-building", "title": "House building", "text": "

    Build new houses when you're 2 of population down to the limit

    "}, {"location": "age_of_empires/#nice-games", "title": "Nice games", "text": ""}, {"location": "age_of_empires/#tournaments", "title": "Tournaments", "text": ""}, {"location": "age_of_empires/#showmatches", "title": "Showmatches", "text": ""}, {"location": "age_of_empires/#1vs1-games", "title": "1vs1 games", "text": ""}, {"location": "age_of_empires/#references", "title": "References:", "text": ""}, {"location": "aiohttp/", "title": "aiohttp", "text": "

    aiohttp is an asynchronous HTTP Client/Server for asyncio and Python. Think of it as the requests for asyncio.

    "}, {"location": "aiohttp/#installation", "title": "Installation", "text": "
    pip install aiohttp\n

    aiohttp can be bundled with optional libraries to speed up the DNS resolving and other niceties, install it with:

    pip install aiohttp[speedups]\n

    Beware though that some of them don't yet support Python 3.10+

    "}, {"location": "aiohttp/#usage", "title": "Usage", "text": ""}, {"location": "aiohttp/#basic-example", "title": "Basic example", "text": "
    import aiohttp\nimport asyncio\n\nasync def main():\n\n    async with aiohttp.ClientSession() as session:\n        async with session.get('http://python.org') as response:\n\n            print(\"Status:\", response.status)\n            print(\"Content-type:\", response.headers['content-type'])\n\n            html = await response.text()\n            print(\"Body:\", html[:15], \"...\")\n\nloop = asyncio.get_event_loop()\nloop.run_until_complete(main())\n

    This prints:

    Status: 200\nContent-type: text/html; charset=utf-8\nBody: <!doctype html> ...\n

    Why so many lines of code? Check it out here.

    "}, {"location": "aiohttp/#make-a-request", "title": "Make a request", "text": "

    With the snippet from the basic example we have a ClientSession called session and a ClientResponse object called response.

    In order to make an HTTP POST request use ClientSession.post() coroutine:

    session.post('http://httpbin.org/post', data=b'data')\n

    Other HTTP methods are available as well:

    session.put('http://httpbin.org/put', data=b'data')\nsession.delete('http://httpbin.org/delete')\nsession.head('http://httpbin.org/get')\nsession.options('http://httpbin.org/get')\nsession.patch('http://httpbin.org/patch', data=b'data')\n

    To make several requests to the same site more simple, the parameter base_url of ClientSession constructor can be used. For example to request different endpoints of http://httpbin.org can be used the following code:

    async with aiohttp.ClientSession('http://httpbin.org') as session:\n    async with session.get('/get'):\n        pass\n    async with session.post('/post', data=b'data'):\n        pass\n    async with session.put('/put', data=b'data'):\n        pass\n

    Use the response.raise_for_status() method to raise an exception if the status code is higher than 400.

    "}, {"location": "aiohttp/#passing-parameters-in-urls", "title": "Passing parameters in urls", "text": "

    You often want to send some sort of data in the URL\u2019s query string. If you were constructing the URL by hand, this data would be given as key/value pairs in the URL after a question mark, e.g. httpbin.org/get?key=val. Requests allows you to provide these arguments as a dict, using the params keyword argument. As an example, if you wanted to pass key1=value1 and key2=value2 to httpbin.org/get, you would use the following code:

    params = {'key1': 'value1', 'key2': 'value2'}\nasync with session.get('http://httpbin.org/get',\n                       params=params) as resp:\n    expect = 'http://httpbin.org/get?key1=value1&key2=value2'\n    assert str(resp.url) == expect\n

    You can see that the URL has been correctly encoded by printing the URL.

    "}, {"location": "aiohttp/#passing-a-json-in-the-request", "title": "Passing a json in the request", "text": "

    There\u2019s also a built-in JSON decoder, in case you\u2019re dealing with JSON data:

    async with session.get('https://api.github.com/events') as resp:\n    print(await resp.json())\n

    In case that JSON decoding fails, json() will raise an exception.

    "}, {"location": "aiohttp/#setting-custom-headers", "title": "Setting custom headers", "text": "

    If you need to add HTTP headers to a request, pass them in a dict to the headers parameter.

    For example, if you want to specify the content-type directly:

    url = 'http://example.com/image'\npayload = b'GIF89a\\x01\\x00\\x01\\x00\\x00\\xff\\x00,\\x00\\x00'\n          b'\\x00\\x00\\x01\\x00\\x01\\x00\\x00\\x02\\x00;'\nheaders = {'content-type': 'image/gif'}\n\nawait session.post(url,\n                   data=payload,\n                   headers=headers)\n

    You also can set default headers for all session requests:

    headers={\"Authorization\": \"Basic bG9naW46cGFzcw==\"}\nasync with aiohttp.ClientSession(headers=headers) as session:\n    async with session.get(\"http://httpbin.org/headers\") as r:\n        json_body = await r.json()\n        assert json_body['headers']['Authorization'] == \\\n            'Basic bG9naW46cGFzcw=='\n

    Typical use case is sending JSON body. You can specify content type directly as shown above, but it is more convenient to use special keyword json:

    await session.post(url, json={'example': 'text'})\n

    For text/plain

    await session.post(url, data='\u041f\u0440\u0438\u0432\u0435\u0442, \u041c\u0438\u0440!')\n
    "}, {"location": "aiohttp/#set-custom-cookies", "title": "Set custom cookies", "text": "

    To send your own cookies to the server, you can use the cookies parameter of ClientSession constructor:

    url = 'http://httpbin.org/cookies'\ncookies = {'cookies_are': 'working'}\nasync with ClientSession(cookies=cookies) as session:\n    async with session.get(url) as resp:\n        assert await resp.json() == {\n           \"cookies\": {\"cookies_are\": \"working\"}}\n
    "}, {"location": "aiohttp/#proxy-support", "title": "Proxy support", "text": "

    aiohttp supports plain HTTP proxies and HTTP proxies that can be upgraded to HTTPS via the HTTP CONNECT method. To connect, use the proxy parameter:

    async with aiohttp.ClientSession() as session:\n    async with session.get(\"http://python.org\",\n                           proxy=\"http://proxy.com\") as resp:\n        print(resp.status)\n

    It also supports proxy authorization:

    async with aiohttp.ClientSession() as session:\n    proxy_auth = aiohttp.BasicAuth('user', 'pass')\n    async with session.get(\"http://python.org\",\n                           proxy=\"http://proxy.com\",\n                           proxy_auth=proxy_auth) as resp:\n        print(resp.status)\n

    Authentication credentials can be passed in proxy URL:

    session.get(\"http://python.org\",\n            proxy=\"http://user:pass@some.proxy.com\")\n

    Contrary to the requests library, it won\u2019t read environment variables by default. But you can do so by passing trust_env=True into aiohttp.ClientSession constructor for extracting proxy configuration from HTTP_PROXY, HTTPS_PROXY, WS_PROXY or WSS_PROXY environment variables (all are case insensitive):

    async with aiohttp.ClientSession(trust_env=True) as session:\n    async with session.get(\"http://python.org\") as resp:\n        print(resp.status)\n
    "}, {"location": "aiohttp/#how-to-use-the-clientsession", "title": "How to use the ClientSession", "text": "

    By default the aiohttp.ClientSession object will hold a connector with a maximum of 100 connections, putting the rest in a queue. This is quite a big number, this means you must be connected to a hundred different servers (not pages!) concurrently before even having to consider if your task needs resource adjustment.

    In fact, you can picture the session object as a user starting and closing a browser: it wouldn\u2019t make sense to do that every time you want to load a new tab.

    So you are expected to reuse a session object and make many requests from it. For most scripts and average-sized software, this means you can create a single session, and reuse it for the entire execution of the program. You can even pass the session around as a parameter in functions. For example, the typical \u201chello world\u201d:

    import aiohttp\nimport asyncio\n\nasync def main():\n    async with aiohttp.ClientSession() as session:\n        async with session.get('http://python.org') as response:\n            html = await response.text()\n            print(html)\n\nloop = asyncio.get_event_loop()\nloop.run_until_complete(main())\n

    Can become this:

    import aiohttp\nimport asyncio\n\nasync def fetch(session, url):\n    async with session.get(url) as response:\n        return await response.text()\n\nasync def main():\n    async with aiohttp.ClientSession() as session:\n        html = await fetch(session, 'http://python.org')\n        print(html)\n\nloop = asyncio.get_event_loop()\nloop.run_until_complete(main())\n

    When to create more than one session object then? It arises when you want more granularity with your resources management:

    "}, {"location": "aiohttp/#an-aiohttp-adapter", "title": "An aiohttp adapter", "text": "
    import asyncio\nimport aiohttp\nimport json\n\nfrom dataclasses import dataclass\n\n\n@dataclass\nclass Config:\n    verify_ssl: bool = True\n    tcp_connections: int = 5\n\n\nclass Http:\n    \"\"\"A generic HTTP Rest adapter.\"\"\"\n\n    def __init__(self, config: Optional[Config] = None) -> None:\n        self.config = Config() if config is None else config\n\n    async def __aenter__(self) -> 'Http':\n        self._con = aiohttp.TCPConnector(\n            verify_ssl=self.config.verify_ssl, limit=self.config.tcp_connections\n        )\n        self._session = aiohttp.ClientSession(connector=self._con)\n        return self\n\n    async def __aexit__(self, exc_type, exc, tb) -> None:\n        await self._session.close()\n        await self._con.close()\n\n    async def request(\n        self,\n        url: str,\n        method: str = 'get',\n        query_param: Optional[Dict] = None,\n        headers: Optional[Dict] = None,\n        body: Optional[Dict] = None,\n    ) -> aiohttp.ClientResponse:\n        \"\"\"Performs an Async HTTP request.\n\n        Args:\n            method (str): request method ('GET', 'POST', 'PUT', ).\n            url (str): request url.\n            query_param (dict or None): url query parameters.\n            header (dict or None): request headers.\n            body (json or None): request body in case of method POST or PUT.\n        \"\"\"\n        method = method.upper()\n        headers = headers or {}\n\n        if method == \"GET\":\n            log.debug(f\"Fetching page {url}\")\n            async with self._session.get(\n                url, params=query_param, headers=headers\n            ) as response:\n                if response.status != 200:\n                    log.debug(f\"{url} returned an {response.status} code\")\n                    response.raise_for_status()\n                return response\n\n\nasync def main():\n    async with Http() as client:\n        print(await client.request(method=\"GET\", url=\"https://httpbin.org/get\"))\n\n\nif __name__ == \"__main__\":\n    asyncio.run(main) \n
    "}, {"location": "aiohttp/#why-so-many-lines-of-code", "title": "Why so many lines of code", "text": "

    The first time you use aiohttp, you\u2019ll notice that a simple HTTP request is performed not with one, but with up to three steps:

    async with aiohttp.ClientSession() as session:\n    async with session.get('http://python.org') as response:\n        print(await response.text())\n

    It\u2019s especially unexpected when coming from other libraries such as the very popular requests, where the \u201chello world\u201d looks like this:

    response = requests.get('http://python.org')\nprint(response.text)\n

    So why is the aiohttp snippet so verbose?

    Because aiohttp is asynchronous, its API is designed to make the most out of non-blocking network operations. In code like this, requests will block three times, and does it transparently, while aiohttp gives the event loop three opportunities to switch context:

    "}, {"location": "aiohttp/#references", "title": "References", "text": ""}, {"location": "aleph/", "title": "Aleph", "text": "

    Aleph is a tool for indexing large amounts of both documents (PDF, Word, HTML) and structured (CSV, XLS, SQL) data for easy browsing and search. It is built with investigative reporting as a primary use case. Aleph allows cross-referencing mentions of well-known entities (such as people and companies) against watchlists, e.g. from prior research or public datasets.

    "}, {"location": "aleph/#install-the-development-environment", "title": "Install the development environment", "text": "

    As a first step, check out the source code of Aleph from GitHub:

    git clone https://github.com/alephdata/aleph.git\ncd aleph/\n

    Also, please execute the following command to allow ElasticSearch to map its memory:

    sysctl -w vm.max_map_count=262144\n

    Then enable the use of pdb by adding the next lines into the docker-compose.dev.yml file, under the api service configuration.

    stdin_open: true\ntty: true\n

    With the settings in place, you can use make all to set everything up and launch the web service. This is equivalent to the following steps:

    Open http://localhost:8080/ in your browser to visit the web frontend.

    "}, {"location": "aleph/#debugging-the-code", "title": "Debugging the code", "text": "

    To debug the code, you can create pdb breakpoints in the code you cloned, and run the actions that trigger the breakpoint. To be able to act on it, you need to be attached to the api by running:

    docker attach aleph_api_1\n

    You don't need to reload the page for it to load the changes, it does it dynamically.

    "}, {"location": "aleph/#operation", "title": "Operation", "text": ""}, {"location": "aleph/#upgrade-aleph", "title": "Upgrade Aleph", "text": "

    Aleph does not perform updates and database migrations automatically. Once you have the latest version, you can run the command bellow to upgrade the existing installation (i.e. apply changes to the database model or the search index format).

    The first step is to add a notice in Aleph's banner section informing the users that there's going to be a downtime.

    Before you upgrade, check the to make sure you understand the latest release and know about new options and features that have been added.

    In production mode, make sure you perform a backup of the main database and the ElasticSearch index before running an upgrade.

    Then, make sure you are using the latest docker-compose.yml file. You can do this by checking out the source repo, but really you just need that one file (and your config in aleph.env). There are many docker-compose.yml files, we need to decide the one we want to take as the source of truth.

    # Pull changes\ncd /data/config/aleph\ndocker-compose pull --parallel\n\n# Stop services\nservice aleph stop\n\n# Do database migrations\ndocker-compose up -d redis postgres elasticsearch\n# Wait a minute or so while services boot up...\n# Run upgrade:\ndocker-compose run --rm shell aleph upgrade\n\n# Start the services\nservice aleph start\n
    "}, {"location": "aleph/#create-aleph-admins", "title": "Create Aleph admins", "text": "

    Creation of admins depends on how you create users, in our case that we're using Oauth we need to update the database directly (ugly!). So go into the instance you want to do the change and run:

    # Create a terminal inside the aleph environment\ndocker-compose run --rm shell bash\n\n# Connect to the postgres database\n# It will ask you for the password, search it in the docker-compose.yaml file\npsql postgresql://aleph@postgres/aleph\n\n# Once there you can see which users do have the Admin rights with:\nselect * from role where is_admin = 't';\n\n# If you want to make another user admin run the next command:\nUPDATE role SET is_admin = true WHERE email = 'keng@pulitzercenter.org';\n

    You may also need to run aleph update afterwards to refresh some cached information.

    "}, {"location": "aleph/#remove-a-group", "title": "Remove a group", "text": "

    There is currently no web interface that allows this operation, you need to interact with the database directly.

    # Create a terminal inside the aleph environment\ndocker-compose run --rm shell bash\n\n# Connect to the postgres database\n# It will ask you for the password, search it in the docker-compose.yaml file\npsql postgresql://aleph@postgres/aleph\n\n# List the available groups\nselect * from role where type = 'group';\n\n# Delete a group.\n# Imagine that the id of the group we want to delete is 18\ndelete from role where id = 18;\n
    "}, {"location": "aleph/#role-permission-error", "title": "Role permission error", "text": "

    You may encounter the next error:

    ERROR:  update or delete on table \"role\" violates foreign key constraint \"permission_role_id_fkey\" on table \"permission\"\nDETAIL:  Key (id)=(18) is still referenced from table \"permission\".\n

    That means that the group is still used, to find who is using it use:

    select * from permission where role_id = 18;\n

    You can check the elements that have the permission by looking at the collection_id number, imagine it's 3, then you can check your.url.com/investigations/3.

    Once you're sure you can remove that permission, run:

    delete from permission where role_id = 18;\ndelete from role where id = 18;\n
    "}, {"location": "aleph/#role-membership-error", "title": "Role membership error", "text": "

    You may encounter the next error:

    ERROR:  update or delete on table \"role\" violates foreign key constraint \"role_membership_group_id_fkey\" on table \"role_membership\"\nDETAIL:  Key (id)=(8) is still referenced from table \"role_membership\".\n
    That means that the group is still used, to find who is using it use:

    select * from role_membership where group_id = 8;\n

    If you agree to remove that user from the group use:

    delete from role_membership where group_id = 8;\ndelete from role where id = 8;\n
    "}, {"location": "aleph/#troubleshooting", "title": "Troubleshooting", "text": ""}, {"location": "aleph/#ingest-gets-stuck", "title": "Ingest gets stuck", "text": "

    It looks that Aleph doesn't yet give an easy way to debug it. It can be seen in the next webs:

    Some interesting ideas I've extracted while diving into these issues is that:

    There are some tickets that attempt to address these issues on the command line:

    I think it's interesting either to contribute to alephclient to solve those issues or if it's complicated create a small python script to detect which files were not uploaded and try to reindex them and/or open issues that will prevent future ingests to fail.

    "}, {"location": "aleph/#problems-accessing-redis-locally", "title": "Problems accessing redis locally", "text": "

    If you're with the VPN connected, turn it off.

    "}, {"location": "aleph/#pdb-behaves-weird", "title": "PDB behaves weird", "text": "

    Sometimes you have two traces at the same time, so each time you run a PDB command it jumps from pdb trace. Quite confusing. Try to c the one you don't want so that you're left with the one you want. Or put the pdb trace in a conditional that only matches one of both threads.

    "}, {"location": "aleph/#references", "title": "References", "text": ""}, {"location": "alot/", "title": "alot", "text": "

    alot is a terminal-based mail user agent based on the notmuch mail indexer. It is written in python using the urwid toolkit and features a modular and command prompt driven interface to provide a full MUA experience.

    "}, {"location": "alot/#installation", "title": "Installation", "text": "
    sudo apt-get install alot\n
    "}, {"location": "alot/#configuration", "title": "Configuration", "text": "

    Alot reads the INI config file ~/.config/alot/config. That file is not created by default, if you don't want to start from scratch, you can use pazz's alot configuration, in particular the [accounts] section.

    "}, {"location": "alot/#ui-interaction", "title": "UI interaction", "text": "

    Basic movement is done with:

    The interface shows one buffer at a time, basic buffer management is done with:

    The buffer type or mode (displayed at the bottom left) determines which prompt commands are available. Usage information on any command can be listed by typing help YOURCOMMAND to the prompt. The key bindings for the current mode are listed upon pressing ?.

    You can always run commands with :.

    "}, {"location": "alot/#troubleshooting", "title": "Troubleshooting", "text": ""}, {"location": "alot/#remove-emails", "title": "Remove emails", "text": "

    Say you want to remove emails from the provider's server but keep them in the notmuch database. There is no straight way to do it, you need to tag them with a special tag like deleted and then remove them from the server with a post-hook.

    "}, {"location": "alot/#theme-not-found", "title": "Theme not found", "text": "

    I don't know why but apt-get didn't install the default themes, you need to create the ~/.config/alot/themes and copy the contents of the themes directory.

    "}, {"location": "alot/#references", "title": "References", "text": ""}, {"location": "amazfit_band_5/", "title": "Amazfit Band 5", "text": "

    Amazfit Band 5 it's the affordable fitness tracker I chose to buy because:

    "}, {"location": "amazfit_band_5/#sleep-detection-quality", "title": "Sleep detection quality", "text": "

    The sleep tracking using Gadgetbridge is not good at all. After two nights, the band has not been able to detect when I woke in the middle of the night, or when I really woke up, as I usually stay in the bed for a time before standing up. I'll try with the proprietary application soon and compare results.

    If it doesn't work either, I might think of getting a specific device like withings sleep analyzer which seems to have much more accuracy and useful insights. I've sent them an email to see if it's possible to extract the data before it reach their servers, and they confirmed that there is no way. Maybe you can route the requests to their servers to one of your own, bring up an http server and reverse engineer the communication.

    Karlicoss, the author of the awesome HPI uses the Emfit QS, so that could be another option.

    "}, {"location": "amazfit_band_5/#firmware-updates", "title": "Firmware updates", "text": "

    Gadgetbridge people have a guide on how to upgrade the firmware, you need to get the firmware from the geek doing forum though, so it is interesting to create an account and watch the post.

    "}, {"location": "android_tips/", "title": "Android tips", "text": ""}, {"location": "android_tips/#extend-the-life-of-your-battery", "title": "Extend the life of your battery", "text": "

    Research has shown that keeping your battery charged between 0% and 80% can make your battery's lifespan last 2x longer than when you use a full battery cycle from 0-100%.

    As a non root user you can install Accubattery (not in F-droid :( ) to get an alarm when the battery reaches 80% so that you can manually unplug it. Instead of leaving the mobile charge in the night and stay connected at 100% a lot of hours until you unplug, charge it throughout the day.

    "}, {"location": "anki/", "title": "Anki", "text": "

    Anki is a program which makes remembering things easy through spaced repetition. Because it's a lot more efficient than traditional study methods, you can either greatly decrease your time spent studying, or greatly increase the amount you learn.

    Anyone who needs to remember things in their daily life can benefit from Anki. Since it is content-agnostic and supports images, audio, videos and scientific markup (via LaTeX), the possibilities are endless.

    "}, {"location": "anki/#installation", "title": "Installation", "text": "

    Install the dependencies:

    sudo apt-get install zstd\n

    Download the latest release package.

    Open a terminal and run the following commands, replacing the filename as appropriate:

    tar xaf Downloads/anki-2.1.XX-linux-qt6.tar.zst\ncd anki-2.1.XX-linux-qt6\nsudo ./install.sh\n
    "}, {"location": "anki/#anki-workflow", "title": "Anki workflow", "text": ""}, {"location": "anki/#how-long-to-do-study-sessions", "title": "How long to do study sessions", "text": "

    I have two study modes:

    The relief thought you can have is that as long as you keep a steady pace of 10/20 mins each day, inevitably you'll eventually finish your pending cards as you're more effective reviewing cards than entering new ones

    "}, {"location": "anki/#what-to-do-with-hard-cards", "title": "What to do with \"hard\" cards", "text": "

    If you're afraid to be stuck in a loop of reviewing \"hard\" cards, don't be. In reality after you've seen that \"hard\" card three times in a row you won't mark it as hard again, because you will remember. If you don't maybe there are two reasons:

    "}, {"location": "anki/#what-to-do-with-unneeded-cards", "title": "What to do with unneeded cards", "text": "

    You have three options:

    Unless you're certain that you are not longer going to need it, suspend it.

    "}, {"location": "anki/#interacting-with-python", "title": "Interacting with python", "text": ""}, {"location": "anki/#configuration", "title": "Configuration", "text": "

    Although there are some python libraries:

    I think the best way is to use AnkiConnect

    The installation process is similar to other Anki plugins and can be accomplished in three steps:

    Anki must be kept running in the background in order for other applications to be able to use Anki-Connect. You can verify that Anki-Connect is running at any time by accessing localhost:8765 in your browser. If the server is running, you will see the message Anki-Connect displayed in your browser window.

    "}, {"location": "anki/#usage", "title": "Usage", "text": "

    Every request consists of a JSON-encoded object containing an action, a version, contextual params, and a key value used for authentication (which is optional and can be omitted by default). Anki-Connect will respond with an object containing two fields: result and error. The result field contains the return value of the executed API, and the error field is a description of any exception thrown during API execution (the value null is used if execution completed successfully).

    Sample successful response:

    {\"result\": [\"Default\", \"Filtered Deck 1\"], \"error\": null}\n

    Samples of failed responses:

    {\"result\": null, \"error\": \"unsupported action\"}\n\n{\"result\": null, \"error\": \"guiBrowse() got an unexpected keyword argument 'foobar'\"}\n

    For compatibility with clients designed to work with older versions of Anki-Connect, failing to provide a version field in the request will make the version default to 4.

    To make the interaction with the API easier, I'm using the next adapter:

    class Anki:\n    \"\"\"Define the Anki adapter.\"\"\"\n\n    def __init__(self, url: str = \"http://localhost:8765\") -> None:\n        \"\"\"Initialize the adapter.\"\"\"\n        self.url = url\n\n    def requests(\n        self, action: str, params: Optional[Dict[str, str]] = None\n    ) -> Response:\n        \"\"\"Do a request to the server.\"\"\"\n        if params is None:\n            params = {}\n\n        response = requests.post(\n            self.url, json={\"action\": action, \"params\": params, \"version\": 6}\n        ).json()\n        if len(response) != 2:\n            raise Exception(\"response has an unexpected number of fields\")\n        if \"error\" not in response:\n            raise Exception(\"response is missing required error field\")\n        if \"result\" not in response:\n            raise Exception(\"response is missing required result field\")\n        if response[\"error\"] is not None:\n            raise Exception(response[\"error\"])\n        return response[\"result\"]\n

    You can find the full adapter in the fala project.

    "}, {"location": "anki/#decks", "title": "Decks", "text": ""}, {"location": "anki/#get-all-decks", "title": "Get all decks", "text": "

    With the adapter:

    self.requests(\"deckNames\")\n

    Or with curl:

    curl localhost:8765 -X POST -d '{\"action\": \"deckNames\", \"version\": 6}'\n
    "}, {"location": "anki/#create-a-new-deck", "title": "Create a new deck", "text": "
    self.requests(\"createDeck\", {\"deck\": deck})\n
    "}, {"location": "anki/#configure-self-hosted-synchronization", "title": "Configure self hosted synchronization", "text": "

    NOTE: In the end I dropped this path and used Ankidroid alone with syncthing as I didn't need to interact with the decks from the computer. Also the ecosystem of synchronization in Anki at 2023-11-10 is confusing as there are many servers available, not all are compatible with the clients and Anki itself has released it's own so some of the community ones will eventually die.

    "}, {"location": "anki/#install-the-server", "title": "Install the server", "text": "

    I'm going to install anki-sync-server as it's simpler to djankiserv:

    # make sure that your dns has a cname set for anki and that your anki container is not using a base url\n\nserver {\n  listen 443 ssl;\n  listen [::]:443 ssl;\n\n  server_name anki.*;\n\n  include /config/nginx/ssl.conf;\n\n  client_max_body_size 0;\n\n  # enable for ldap auth, fill in ldap details in ldap.conf\n  #include /config/nginx/ldap.conf;\n\n  location / {\n      # enable the next two lines for http auth\n      #auth_basic \"Restricted\";\n      #auth_basic_user_file /config/nginx/.htpasswd;\n\n      # enable the next two lines for ldap auth\n      #auth_request /auth;\n      #error_page 401 =200 /login;\n\n      include /config/nginx/proxy.conf;\n      resolver 127.0.0.11 valid=30s;\n      set $upstream_anki anki;\n      proxy_pass http://$upstream_anki:27701;\n  }\n}\n

    ankisyncctl.py has more commands to manage your users:

    "}, {"location": "anki/#configure-ankidroid", "title": "Configure AnkiDroid", "text": ""}, {"location": "anki/#configure-anki", "title": "Configure Anki", "text": "

    Install addon from ankiweb (support 2.1)

    "}, {"location": "anki/#references", "title": "References", "text": ""}, {"location": "anonymous_feedback/", "title": "Anonymous Feedback", "text": "

    Anonymous Feedback is a communication tool where people share feedback to teammates or other organizational members while protecting their identities.

    "}, {"location": "anonymous_feedback/#why-would-you-need-anonymous-feedback", "title": "Why would you need anonymous feedback?", "text": "

    Ideally, everyone in your company should be able to give feedback publicly and not anonymously. They should share constructive criticism and not shy away from direct feedback if they believe and trust that their opinions will be heard and addressed.

    However, to achieve this ideal, people need to feel that they are in a safe space, a place or environment in which they feel confident that they will not be exposed to discrimination, criticism, harassment, or any other emotional or physical harm. The work place is usually not considered a safe space by the employees because they may:

    For all these reasons, some employees may remain silent when asked for direct feedback, to speak up against an internal issue or in need to report a colleague or manager. These factors are further amplified if:

    Until the safe space is built where direct feedback is viable, anonymous feedback gives these employees a mechanism to raise their concerns, practice their feedback-giving skills, test the waters, and understand how people perceive their constructive (and sometimes critical) opinions, thus building the needed trust.

    "}, {"location": "anonymous_feedback/#pros-and-cons", "title": "Pros and cons", "text": "

    Pros of Anonymous Feedback:

    Cons of Anonymous Feedback:

    "}, {"location": "anonymous_feedback/#how-to-request-anonymous-feedback", "title": "How to request anonymous feedback", "text": "

    When requesting for anonymous feedback on an organizational level, it is necessary to:

    "}, {"location": "anonymous_feedback/#how-to-act-on-anonymous-feedback", "title": "How to Act on Anonymous Feedback", "text": "

    Once you have sent the anonymous feedback, be sure to:

    "}, {"location": "anonymous_feedback/#references", "title": "References", "text": ""}, {"location": "ansible_snippets/", "title": "Ansible Snippets", "text": ""}, {"location": "ansible_snippets/#ansible-retry-a-failed-job", "title": "Ansible retry a failed job", "text": "
    - command: /usr/bin/false\n  retries: 3\n  delay: 3\n  register: result\n  until: result.rc == 0\n
    "}, {"location": "ansible_snippets/#ansible-add-a-sleep", "title": "Ansible add a sleep", "text": "
    - name: Pause for 5 minutes to build app cache\n  ansible.builtin.pause:\n    minutes: 5\n
    "}, {"location": "ansible_snippets/#ansible-condition-that-uses-a-regexp", "title": "Ansible condition that uses a regexp", "text": "
    - name: Check if an instance name or hostname matches a regex pattern\n  when: inventory_hostname is not match('molecule-.*')\n  fail:\n    msg: \"not a molecule instance\"\n
    "}, {"location": "ansible_snippets/#ansible-lint-doesnt-find-requirements", "title": "Ansible-lint doesn't find requirements", "text": "

    It may be because you're using requirements.yaml instead of requirements.yml. Create a temporal link from one file to the other, run the command and then remove the link.

    It will work from then on even if you remove the link. \u00af\\(\u00b0_o)/\u00af

    "}, {"location": "ansible_snippets/#run-task-only-once", "title": "Run task only once", "text": "

    Add run_once: true on the task definition:

    - name: Do a thing on the first host in a group.\n  debug: \n    msg: \"Yay only prints once\"\n  run_once: true\n
    "}, {"location": "ansible_snippets/#run-command-on-a-working-directory", "title": "Run command on a working directory", "text": "
    - name: Change the working directory to somedir/ and run the command as db_owner \n  ansible.builtin.command: /usr/bin/make_database.sh db_user db_name\n  become: yes\n  become_user: db_owner\n  args:\n    chdir: somedir/\n    creates: /path/to/database\n
    "}, {"location": "ansible_snippets/#run-handlers-in-the-middle-of-the-tasks-file", "title": "Run handlers in the middle of the tasks file", "text": "

    If you need handlers to run before the end of the play, add a task to flush them using the meta module, which executes Ansible actions:

    tasks:\n  - name: Some tasks go here\n    ansible.builtin.shell: ...\n\n  - name: Flush handlers\n    meta: flush_handlers\n\n  - name: Some other tasks\n    ansible.builtin.shell: ...\n

    The meta: flush_handlers task triggers any handlers that have been notified at that point in the play.

    Once handlers are executed, either automatically after each mentioned section or manually by the flush_handlers meta task, they can be notified and run again in later sections of the play.

    "}, {"location": "ansible_snippets/#run-command-idempotently", "title": "Run command idempotently", "text": "
    - name: Register the runner in gitea\n  become: true\n  command: act_runner register --config config.yaml --no-interactive --instance {{ gitea_url }} --token {{ gitea_docker_runner_token }}\n  args:\n    creates: /var/lib/gitea_docker_runner/.runner\n
    "}, {"location": "ansible_snippets/#get-the-correct-architecture-string", "title": "Get the correct architecture string", "text": "

    If you have an amd64 host you'll get x86_64, but sometimes you need the amd64 string. On those cases you can use the next snippet:

    ---\n# vars/main.yaml\ndeb_architecture: \n  aarch64: arm64\n  x86_64: amd64\n\n---\n# tasks/main.yaml\n- name: Download the act runner binary\n  become: True\n  ansible.builtin.get_url:\n    url: https://dl.gitea.com/act_runner/act_runner-linux-{{ deb_architecture[ansible_architecture] }}\n    dest: /usr/bin/act_runner\n    mode: '0755'\n
    "}, {"location": "ansible_snippets/#check-the-instances-that-are-going-to-be-affected-by-playbook-run", "title": "Check the instances that are going to be affected by playbook run", "text": "

    Useful to list the instances of a dynamic inventory

    ansible-inventory -i aws_ec2.yaml --list\n
    "}, {"location": "ansible_snippets/#check-if-variable-is-defined-or-empty", "title": "Check if variable is defined or empty", "text": "

    In Ansible playbooks, it is often a good practice to test if a variable exists and what is its value.

    Particularity this helps to avoid different \u201cVARIABLE IS NOT DEFINED\u201d errors in Ansible playbooks.

    In this context there are several useful tests that you can apply using Jinja2 filters in Ansible.

    "}, {"location": "ansible_snippets/#check-if-ansible-variable-is-defined-exists", "title": "Check if Ansible variable is defined (exists)", "text": "
    tasks:\n\n- shell: echo \"The variable 'foo' is defined: '{{ foo }}'\"\n  when: foo is defined\n\n- fail: msg=\"The variable 'bar' is not defined\"\n  when: bar is undefined\n
    "}, {"location": "ansible_snippets/#check-if-ansible-variable-is-empty", "title": "Check if Ansible variable is empty", "text": "
    tasks:\n\n- fail: msg=\"The variable 'bar' is empty\"\n  when: bar|length == 0\n\n- shell: echo \"The variable 'foo' is not empty: '{{ foo }}'\"\n  when: foo|length > 0\n
    "}, {"location": "ansible_snippets/#check-if-ansible-variable-is-defined-and-not-empty", "title": "Check if Ansible variable is defined and not empty", "text": "
    tasks:\n\n- shell: echo \"The variable 'foo' is defined and not empty\"\n  when: (foo is defined) and (foo|length > 0)\n\n- fail: msg=\"The variable 'bar' is not defined or empty\"\n  when: (bar is not defined) or (bar|length == 0)\n
    "}, {"location": "ansible_snippets/#start-and-enable-a-systemd-service", "title": "Start and enable a systemd service", "text": "

    Typically defined in handlers/main.yaml:

    - name: Restart the service\n  become: true\n  systemd:\n    name: zfs_exporter\n    enabled: true\n    daemon_reload: true\n    state: started\n

    And used in any task:

    - name: Create the systemd service\n  become: true\n  template:\n    src: service.j2\n    dest: /etc/systemd/system/zfs_exporter.service\n  notify: Restart the service\n
    "}, {"location": "ansible_snippets/#download-a-file", "title": "Download a file", "text": "
    - name: Download foo.conf\n  ansible.builtin.get_url:\n    url: http://example.com/path/file.conf\n    dest: /etc/foo.conf\n    mode: '0440'\n
    "}, {"location": "ansible_snippets/#download-an-decompress-a-targz", "title": "Download an decompress a tar.gz", "text": "
    - name: Unarchive a file that needs to be downloaded (added in 2.0)\n  ansible.builtin.unarchive:\n    src: https://example.com/example.zip\n    dest: /usr/local/bin\n    remote_src: yes\n

    If you want to only extract a file you can use the includes arg

    - name: Download the zfs exporter\n  become: true\n  ansible.builtin.unarchive:\n    src: https://github.com/pdf/zfs_exporter/releases/download/v{{ zfs_exporter_version }}/zfs_exporter-{{ zfs_exporter_version }}.linux-amd64.tar.gz\n    dest: /usr/local/bin\n    include: zfs_exporter\n    remote_src: yes\n    mode: 0755\n

    But that snippet sometimes fail, you can alternatively download it locally and copy it:

    - name: Test if zfs_exporter binary exists\n  stat:\n    path: /usr/local/bin/zfs_exporter\n  register: zfs_exporter_binary\n\n- name: Install the zfs exporter\n  block:\n    - name: Download the zfs exporter\n      delegate_to: localhost\n      ansible.builtin.unarchive:\n        src: https://github.com/pdf/zfs_exporter/releases/download/v{{ zfs_exporter_version }}/zfs_exporter-{{ zfs_exporter_version }}.linux-amd64.tar.gz\n        dest: /tmp/\n        remote_src: yes\n\n    - name: Upload the zfs exporter to the server\n      become: true\n      copy:\n        src: /tmp/zfs_exporter-{{ zfs_exporter_version }}.linux-amd64/zfs_exporter\n        dest: /usr/local/bin\n        mode: 0755\n  when: not zfs_exporter_binary.stat.exists\n
    "}, {"location": "ansible_snippets/#skip-ansible-lint-for-some-tasks", "title": "Skip ansible-lint for some tasks", "text": "
    - name: Modify permissions\n  command: >\n    chmod -R g-w /home/user\n  tags:\n    - skip_ansible_lint\n  sudo: yes\n
    "}, {"location": "ansible_snippets/#authorize-an-ssh-key", "title": "Authorize an SSH key", "text": "
    - name: Authorize the sender ssh key\n  authorized_key:\n    user: syncoid\n    state: present\n    key: \"{{ syncoid_receive_ssh_key }}\"\n
    "}, {"location": "ansible_snippets/#create-a-user", "title": "Create a user", "text": "

    The following snippet creates a user with password login disabled.

    - name: Create the syncoid user\n  ansible.builtin.user:\n    name: syncoid\n    state: present\n    password: !\n    shell: /usr/sbin/nologin\n

    If you don't set a password any user can do su your_user to set a random password use the next snippet:

    - name: Create the syncoid user\n  ansible.builtin.user:\n    name: syncoid\n    state: present\n    password: \"{{ lookup('password', '/dev/null', length=50, encrypt='sha512_crypt') }}\"\n    shell: /bin/bash\n

    This won't pass the idempotence tests as it doesn't save the password anywhere (/dev/null) in the controler machine.

    "}, {"location": "ansible_snippets/#create-an-ssh-key", "title": "Create an ssh key", "text": "
    - name: Create .ssh directory\n  become: true\n  file:\n    path: /root/.ssh\n    state: directory\n    mode: 700\n\n- name: Create the SSH key to directory\n  become: true\n  openssh_keypair:\n    path: /root/.ssh/id_ed25519\n    type: ed25519\n  register: ssh\n\n- name: Show public key\n  debug:\n    var: ssh.public_key\n
    "}, {"location": "ansible_snippets/#get-the-hosts-of-a-dynamic-ansible-inventory", "title": "Get the hosts of a dynamic ansible inventory", "text": "
    ansible-inventory -i environments/production --graph\n

    You can also use the --list flag to get more info of the hosts.

    "}, {"location": "ansible_snippets/#speed-up-the-stat-module", "title": "Speed up the stat module", "text": "

    The stat module calculates the checksum and the md5 of the file in order to get the required data. If you just want to check if the file exists use:

    - name: Verify swapfile status\n  stat:\n    path: \"{{ common_swapfile_location }}\"\n    get_checksum: no\n    get_md5: no\n    get_mime: no\n    get_attributes: no\n  register: swap_status\n  changed_when: not swap_status.stat.exists\n
    "}, {"location": "ansible_snippets/#stop-running-docker-containers", "title": "Stop running docker containers", "text": "
    - name: Get running containers\n  docker_host_info:\n    containers: yes\n  register: docker_info\n\n- name: Stop running containers\n  docker_container:\n    name: \"{{ item }}\"\n    state: stopped\n  loop: \"{{ docker_info.containers | map(attribute='Id') | list }}\"\n
    "}, {"location": "ansible_snippets/#moving-a-file-remotely", "title": "Moving a file remotely", "text": "

    Funnily enough, you can't without a command. You could use the copy module with:

    - name: Copy files from foo to bar\n  copy:\n    remote_src: True\n    src: /path/to/foo\n    dest: /path/to/bar\n\n- name: Remove old files foo\n  file: path=/path/to/foo state=absent\n

    But that doesn't move, it copies and removes, which is not the same.

    To make the command idempotent you can use a stat task before.

    - name: stat foo\n  stat: \n    path: /path/to/foo\n  register: foo_stat\n\n- name: Move foo to bar\n  command: mv /path/to/foo /path/to/bar\n  when: foo_stat.stat.exists\n
    "}, {"location": "anticolonialism/", "title": "Anti-Colonialism", "text": ""}, {"location": "anticolonialism/#references", "title": "References", "text": ""}, {"location": "anticolonialism/#music", "title": "Music", "text": ""}, {"location": "antifascism/", "title": "Antifascism", "text": "

    Antifascism is a method of politics, a locus of individual and group self-indentification, it's a transnational movement that adapted preexisting socialist, anarchist, and communist currents to a sudden need to react to the far right menace (Mark p. 11). It's based on the idea that any oppression form can't be allowed, and should be actively fought with whatever means are necessary. Usually sharing space and even blending with other politic stances that share the same principle, such as intersectional feminism.

    Read the references

    The articles under this section are the brushstrokes I use to learn how to become an efficient antifascist.

    It assumes that you identify yourself as an antifascist, so I'll go straight to the point, skipping much of the argumentation that is needed to sustain these ideas. I'll add links to Mark's and Pol's awesome books, which I strongly recommend you to buy, as they both are jewels that everyone should read.

    Despite the criminalization and stigmatization by the mainstream press and part of the society, antifascism is a rock solid organized movement with a lot of history, that has invested blood, tears and lives to prevent us from living in a yet more horrible world.

    The common stereotype is a small group of leftist young people that confront the nazis on the streets, preventing them from using the public space, and from further organizing through direct action and violence if needed. If you don't identify yourself with this stereotype, don't worry, they are only a small (but essential) part of antifascism, there are so many and diverse ways to be part of the antifascist movement that in fact, everyone can (and should) be an antifascist.

    "}, {"location": "antifascism/#what-is-fascism", "title": "What is fascism", "text": "

    Fascism in Paxton's words is:

    ... a form of political behavior marked by obsessive preoccupation with community decline, humiliation, or victimhood and by compensatory cults of unity, energy, and purity, in which a mass-based party of commited nationalist militians, working in uneasy but effective collaboration with traditional elites, abandons democratic liberties and pursues with redemptive violence and without ethical or legal restrains goals of internal cleansing and external expansion.

    They are nourished by the people's weariness with the corruption and inoperability of the traditional political parties, and the growing fear and anguish of an uncertain economic situation.

    They continuously adapt, redefine and reappropriate concepts under an irreverent, politically incorrect and critical spirit, to spread the old discourse of the privileged against the oppressed.

    They dress themselves as antisystems, pursuing the liberty behind the authority, and accepting the democratic system introducing totalitarianism nuances (Pol p.20).

    "}, {"location": "antifascism/#how-to-identify-fascism", "title": "How to identify fascism", "text": "

    We need to make sure that we use the term well, otherwise we run into the risk of the word loosing meaning. But equally important is not to fall in a wording discussion that paralyzes us.

    One way to make it measurable is to use Kimberl\u00e9 Williams Crenshaw intersectionality theory , which states that individuals experience oppression or privilege based on a belonging to a plurality of social categories, to measure how close an action or discourse follows fascism principles (Pol p.26).

    Source

    Fascism has always been carried out by people with many privileges (the upper part of the diagram) against collectives under many oppressions (the lower part of the diagram). We can then state that the more the oppressions a discourse defends and perpetuates, the more probable it is to be fascist. If it also translates into physical or verbal aggressions, escalates into the will to transform that discourse into laws that backs up those aggressions, or tries to build a government under those ideas, then we clearly have a political roadmap towards fascism.

    The fact that they don't propose to abolish the democracy or try to send people to concentration camps doesn't mean they are not fascist. First, we don't need them to commit the exact same crimes that the fascists of last century made to put at risk some social collectives, and secondly, history tells us that classic fascism movements didn't show their true intentions in their early phases.

    Fascism shifts their form and particular characteristics based on place and time. Waiting to see it clear is risking being late to fight it. Therefore whenever we see a discourse that comes from a privileged person against a oppressed one, we should fight it immediately, once fought, you can analyze if it was fascist or not (Pol p.28)

    "}, {"location": "antifascism/#how-to-fight-fascism", "title": "How to fight fascism", "text": "

    There are many ways to fight it, the book Todo el mundo puede ser Antifa: Manual practico para destruir el fascismo of Pol Andi\u00f1ach gathers some of them.

    One way we've seen pisses them off quite much is when they are ridiculed and they evocate the image of incompetence. It's a fine line to go, because if it falls into a pity image then it may strengthen their victim role.

    "}, {"location": "antifascism/#references", "title": "References", "text": ""}, {"location": "antifascism/#magazines", "title": "Magazines", "text": ""}, {"location": "antifascism/#podcasts", "title": "Podcasts", "text": ""}, {"location": "antifascist_actions/", "title": "Antifa Actions", "text": "

    Collection of amazing and inspiring antifa actions.

    "}, {"location": "antifascist_actions/#2022", "title": "2022", "text": ""}, {"location": "antifascist_actions/#an-open-data-initiative-to-map-spanish-hate-crimes", "title": "An open data initiative to map spanish hate crimes", "text": "

    The project Crimenes de Odio have created an open database of the hate crimes registered in the spanish state.

    "}, {"location": "antifascist_actions/#an-open-data-initiative-to-map-spanish-fascist-icons", "title": "An open data initiative to map spanish fascist icons", "text": "

    The project Deber\u00edaDesaparecer have created an open database of the remains of the spanish fascist regime icons. The visualization they've created is astonishing, and they've provided a form so that anyone can contribute to the dataset.

    "}, {"location": "antifascist_actions/#2021", "title": "2021", "text": ""}, {"location": "antifascist_actions/#a-fake-company-and-five-million-recycled-flyers", "title": "A fake company and five million recycled flyers", "text": "

    A group of artists belonging to the Center for political beauty created a fake company Flyerservice Hahn and convinced more than 80 regional sections of the far right party AfD to hire them to deliver their electoral propaganda.

    They gathered five million flyers, with a total weight of 72 tons. They justify that they wouldn't be able to lie to the people, so they did nothing in the broader sense of the word. They declared that they are the \"world wide leader in the non-delivery of nazi propaganda\". At the start of the electoral campaign, they went to the AfD stands, and they let their members to give them flyers the throw them to the closest bin. \"It's something that any citizen can freely do, we have only industrialized the process\".

    They've done a crowdfunding to fund the legal process that may result.

    "}, {"location": "antitransphobia/", "title": "Anti-transphobia", "text": "

    Anti-transphobia being reductionist is the opposition to the collection of ideas and phenomena that encompass a range of negative attitudes, feelings or actions towards transgender people or transness in general. Transphobia can include fear, aversion, hatred, violence, anger, or discomfort felt or expressed towards people who do not conform to social gender expectations. It is often expressed alongside homophobic views and hence is often considered an aspect of homophobia.

    It's yet another clear case of privileged people oppressing even further already oppressed collectives. We can clearly see it if we use the ever useful Kimberl\u00e9 Williams Crenshaw intersectionality theory diagram.

    Source

    "}, {"location": "antitransphobia/#terf", "title": "TERF", "text": "

    TERF is an acronym for trans-exclusionary radical feminist. The term originally applied to the minority of feminists that expressed transphobic sentiments such as the rejection of the assertion that trans women are women, the exclusion of trans women from women's spaces, and opposition to transgender rights legislation. The meaning has since expanded to refer more broadly to people with trans-exclusionary views who may have no involvement with radical feminism.

    "}, {"location": "antitransphobia/#arguments-against-theories-that-deny-the-reality-of-trans-people", "title": "Arguments against theories that deny the reality of trans people", "text": "

    This section is a direct translation from Alana Portero's text called Definitions.

    "}, {"location": "antitransphobia/#sex-is-a-medical-category-and-gender-a-social-category", "title": "Sex is a medical category and gender a social category", "text": "

    Sex is a medical category, it's not a biological one. According to body features like the chromosome structure and genitalia appearance, medicine assigns the sex (hence gender) to the bodies. In the case of intersexual people, they are usually mutilated and hormonated so that their bodies fit into one of the two options contemplated by the medicine.

    Gender is the term we use to refer to how a person feels about themselves as a boy/man, a girl/woman or non-binary.

    Since birth, we're told what's appropriate (and what isn't) for each gender. These are the gender roles. It's not the same gender than gender role: the gender determines how you interact with the other roles. For example, a woman can take traditionally understood male roles gender roles, that doesn't mean that she is or isn't a woman.

    The problem arises when these two oppressions are mixed up: cissexism (the believe that bodies have an immutable gender defined by the sex assigned at birth) and misogyny (the base of feminine oppression). When you mixing them up you get the idea that the trans movement erases the feminine structural oppression, when in reality, it broadens the scope and makes it more precise, as they suffer the same misogyny than the cis women.

    Women are killed for being women. They are socially assigned the responsibility for care, they are prevented from having individual will and they are deterred from accessing resources. This structural violence is suffered by all women regardless of the sex assigned at birth.

    Questioning the adjudication of gender to the bodies and questioning the roles assigned to the genders are complementary paths for the feminism liberation.

    "}, {"location": "antitransphobia/#avoid-the-interested-manipulation-of-the-sexual-or-gender-identity", "title": "Avoid the interested manipulation of the sexual or gender identity", "text": "

    The sexual or gender identity determines whether there is correspondence with the gender assigned at birth. When there isn't, it concerns a trans person.

    The sex and gender terms represent the same reality, being sex the medical term, and gender the academic one. Equally transexual and transgender represent the same reality, although these last have a pathologizing undertone, the term trans is preferred.

    "}, {"location": "antitransphobia/#avoid-the-fears-of-letting-trans-people-be", "title": "Avoid the fears of letting trans people be", "text": "

    Some are afraid that the trans women negatively affect the statistics of unemployment, laboral inequality, feminization of the poverty and machist violence, and they contradict the problems of the cis women.

    Trans people usually have a greater unemployment rate (85% in Spain), so the glass ceiling is not yet even a concern, and they are also greatly affected by machist violence.

    The queer theory doesn't erase or blur women as a political subject. Thinking that it risks the rights and achievements earned through the feminist movement shows a complete misunderstanding of the theory.

    "}, {"location": "antitransphobia/#women-are-not-an-entity", "title": "Women are not an entity", "text": "

    Women are not an entity, they are a group of people that are placed below men in the social scale, each with her own unique experience. The woman identity belongs to any person that identifies herself with it.

    The fight against discrimination and towards inclusion politics should be mandatory for all society, and shouldn't be used against the trans people.

    "}, {"location": "antitransphobia/#references", "title": "References", "text": ""}, {"location": "argocd/", "title": "ArgoCD", "text": "

    Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes.

    Argo CD follows the GitOps pattern of using Git repositories as the source of truth for defining the desired application state. Kubernetes manifests can be specified in several ways:

    Argo CD automates the deployment of the desired application states in the specified target environments. Application deployments can track updates to branches, tags, or pinned to a specific version of manifests at a Git commit. See tracking strategies for additional details about the different tracking strategies available.

    "}, {"location": "argocd/#using-helmfile", "title": "Using helmfile", "text": "

    helmfile is not yet supported officially, but you can use it through this plugin.

    "}, {"location": "argocd/#references", "title": "References", "text": ""}, {"location": "asyncio/", "title": "Asyncio", "text": "

    asyncio is a library to write concurrent code using the async/await syntax.

    asyncio is used as a foundation for multiple Python asynchronous frameworks that provide high-performance network and web-servers, database connection libraries, distributed task queues, etc.

    asyncio is often a perfect fit for IO-bound and high-level structured network code.

    Note

    \"[Asyncer](https://asyncer.tiangolo.com/tutorial/) looks very useful\"\n
    "}, {"location": "asyncio/#basic-concepts", "title": "Basic concepts", "text": ""}, {"location": "asyncio/#concurrency", "title": "Concurrency", "text": "

    Concurrency is best explained by an example stolen from Miguel Grinberg.

    Chess master Judit Polg\u00e1r hosts a chess exhibition in which she plays multiple amateur players. She has two ways of conducting the exhibition: synchronously and asynchronously.

    Assumptions:

    Synchronous version: Judit plays one game at a time, never two at the same time, until the game is complete. Each game takes (55 + 5) * 30 == 1800 seconds, or 30 minutes. The entire exhibition takes 24 * 30 == 720 minutes, or 12 hours.

    Asynchronous version: Judit moves from table to table, making one move at each table. She leaves the table and lets the opponent make their next move during the wait time. One move on all 24 games takes Judit 24 * 5 == 120 seconds, or 2 minutes. The entire exhibition is now cut down to 120 * 30 == 3600 seconds, or just 1 hour.

    Async IO takes long waiting periods in which functions would otherwise be blocking and allows other functions to run during that downtime. (A function that blocks effectively forbids others from running from the time that it starts until the time that it returns.)

    "}, {"location": "asyncio/#asyncio-is-not-easy", "title": "AsyncIO is not easy", "text": "

    You may have heard the phrase \u201cUse async IO when you can; use threading when you must.\u201d The truth is that building durable multithreaded code can be hard and error-prone. Async IO avoids some of the potential speedbumps that you might otherwise encounter with a threaded design.

    But that\u2019s not to say that async IO in Python is easy. Be warned: when you venture a bit below the surface level, async programming can be difficult too! Python\u2019s async model is built around concepts such as callbacks, events, transports, protocols, and futures\u2014just the terminology can be intimidating. The fact that its API has been changing continually makes it no easier.

    Luckily, asyncio has matured to a point where most of its features are no longer provisional, while its documentation has received a huge overhaul and some quality resources on the subject are starting to emerge as well.

    "}, {"location": "asyncio/#the-asyncawait-syntax-and-native-coroutines", "title": "The async/await Syntax and Native Coroutines", "text": "

    At the heart of async IO are coroutines. A coroutine is a specialized version of a Python generator function. A coroutine is a function that can suspend its execution before reaching return, and it can indirectly pass control to another coroutine for some time. For example look at this Hello World async IO example:

    #!/usr/bin/env python3\n# countasync.py\n\nimport asyncio\n\nasync def count():\n    print(\"One\")\n    await asyncio.sleep(1)\n    print(\"Two\")\n\nasync def main():\n    await asyncio.gather(count(), count(), count())\n\nif __name__ == \"__main__\":\n    import time\n    s = time.perf_counter()\n    asyncio.run(main())\n    elapsed = time.perf_counter() - s\n    print(f\"{__file__} executed in {elapsed:0.2f} seconds.\")\n

    When you execute this file, take note of what looks different than if you were to define the functions with just def and time.sleep():

    $ python3 countasync.py\nOne\nOne\nOne\nTwo\nTwo\nTwo\ncountasync.py executed in 1.01 seconds.\n

    The order of this output is the heart of async IO. Talking to each of the calls to count() is a single event loop, or coordinator. When each task reaches await asyncio.sleep(1), the function talks to the event loop and gives control back to it saying, \u201cI\u2019m going to be sleeping for 1 second. Go ahead and let something else meaningful be done in the meantime.\u201d

    Contrast this to the synchronous version:

    #!/usr/bin/env python3\n# countsync.py\n\nimport time\n\ndef count():\n    print(\"One\")\n    time.sleep(1)\n    print(\"Two\")\n\ndef main():\n    for _ in range(3):\n        count()\n\nif __name__ == \"__main__\":\n    s = time.perf_counter()\n    main()\n    elapsed = time.perf_counter() - s\n    print(f\"{__file__} executed in {elapsed:0.2f} seconds.\")\n

    When executed, there is a slight but critical change in order and execution time:

    $ python3 countsync.py\nOne\nTwo\nOne\nTwo\nOne\nTwo\ncountsync.py executed in 3.01 seconds.\n

    Here time.sleep() can represent any time-consuming blocking function call, while asyncio.sleep() is used to stand in for a non-blocking time-consuming call.

    Summing up the benefit of awaiting something, including asyncio.sleep(), is that the surrounding function can temporarily cede control to another function that\u2019s more readily able to do something immediately. In contrast, time.sleep() or any other blocking call is incompatible with asynchronous Python code, because it will stop everything in its tracks for the duration of the sleep time.

    "}, {"location": "asyncio/#the-rules-of-async-io", "title": "The Rules of Async IO", "text": "

    There are a strict set of rules around when and how you can and cannot use async/await:

    Here are some examples that summarize the above rules:

    async def f(x):\n    y = await z(x)  # OK - `await` and `return` allowed in coroutines\n    return y\n\nasync def g(x):\n    yield x  # OK - this is an async generator\n\nasync def m(x):\n    yield from gen(x)  # No - SyntaxError\n\ndef m(x):\n    y = await z(x)  # No - SyntaxError (no `async def` here)\n    return y\n
    "}, {"location": "asyncio/#async-io-design-patterns", "title": "Async IO Design Patterns", "text": "

    Async IO comes with its own set of possible script designs.

    "}, {"location": "asyncio/#chaining-coroutines", "title": "Chaining Coroutines", "text": "

    This allows you to break programs into smaller, manageable, recyclable coroutines:

    #!/usr/bin/env python3\n# chained.py\n\nimport asyncio\nimport random\nimport time\n\nasync def part1(n: int) -> str:\n    i = random.randint(0, 10)\n    print(f\"part1({n}) sleeping for {i} seconds.\")\n    await asyncio.sleep(i)\n    result = f\"result{n}-1\"\n    print(f\"Returning part1({n}) == {result}.\")\n    return result\n\nasync def part2(n: int, arg: str) -> str:\n    i = random.randint(0, 10)\n    print(f\"part2{n, arg} sleeping for {i} seconds.\")\n    await asyncio.sleep(i)\n    result = f\"result{n}-2 derived from {arg}\"\n    print(f\"Returning part2{n, arg} == {result}.\")\n    return result\n\nasync def chain(n: int) -> None:\n    start = time.perf_counter()\n    p1 = await part1(n)\n    p2 = await part2(n, p1)\n    end = time.perf_counter() - start\n    print(f\"-->Chained result{n} => {p2} (took {end:0.2f} seconds).\")\n\nasync def main(*args):\n    await asyncio.gather(*(chain(n) for n in args))\n\nif __name__ == \"__main__\":\n    import sys\n    random.seed(444)\n    args = [1, 2, 3] if len(sys.argv) == 1 else map(int, sys.argv[1:])\n    start = time.perf_counter()\n    asyncio.run(main(*args))\n    end = time.perf_counter() - start\n    print(f\"Program finished in {end:0.2f} seconds.\")\n

    Pay careful attention to the output, where part1() sleeps for a variable amount of time, and part2() begins working with the results as they become available:

    $ python3 chained.py 9 6 3\npart1(9) sleeping for 4 seconds.\npart1(6) sleeping for 4 seconds.\npart1(3) sleeping for 0 seconds.\nReturning part1(3) == result3-1.\npart2(3, 'result3-1') sleeping for 4 seconds.\nReturning part1(9) == result9-1.\npart2(9, 'result9-1') sleeping for 7 seconds.\nReturning part1(6) == result6-1.\npart2(6, 'result6-1') sleeping for 4 seconds.\nReturning part2(3, 'result3-1') == result3-2 derived from result3-1.\n-->Chained result3 => result3-2 derived from result3-1 (took 4.00 seconds).\nReturning part2(6, 'result6-1') == result6-2 derived from result6-1.\n-->Chained result6 => result6-2 derived from result6-1 (took 8.01 seconds).\nReturning part2(9, 'result9-1') == result9-2 derived from result9-1.\n-->Chained result9 => result9-2 derived from result9-1 (took 11.01 seconds).\nProgram finished in 11.01 seconds.\n

    In this setup, the runtime of main() will be equal to the maximum runtime of the tasks that it gathers together and schedules.

    "}, {"location": "asyncio/#using-a-queue", "title": "Using a Queue", "text": "

    The asyncio package provides queue classes that are designed to be similar to classes of the queue module.

    There is an alternative structure that can also work with async IO: a number of producers, which are not associated with each other, add items to a queue. Each producer may add multiple items to the queue at staggered, random, unannounced times. A group of consumers pull items from the queue as they show up, greedily and without waiting for any other signal.

    In this design, there is no chaining of any individual consumer to a producer. The consumers don\u2019t know the number of producers, or even the cumulative number of items that will be added to the queue, in advance.

    It takes an individual producer or consumer a variable amount of time to put and extract items from the queue, respectively. The queue serves as a throughput that can communicate with the producers and consumers without them talking to each other directly.

    One use-case for queues is for the queue to act as a transmitter for producers and consumers that aren\u2019t otherwise directly chained or associated with each other.

    For example:

    #!/usr/bin/env python3\n# asyncq.py\n\nimport asyncio\nimport itertools \nimport os\nimport random\nimport time\n\nasync def makeitem(size: int = 5) -> str:\n    return os.urandom(size).hex()\n\nasync def randsleep(caller=None) -> None:\n    i = random.randint(0, 10)\n    if caller:\n        print(f\"{caller} sleeping for {i} seconds.\")\n    await asyncio.sleep(i)\n\nasync def produce(name: int, q: asyncio.Queue) -> None:\n    n = random.randint(0, 10)\n    for _ in itertools.repeat(None, n):  # Synchronous loop for each single producer\n        await randsleep(caller=f\"Producer {name}\")\n        i = await makeitem()\n        t = time.perf_counter()\n        await q.put((i, t))\n        print(f\"Producer {name} added <{i}> to queue.\")\n\nasync def consume(name: int, q: asyncio.Queue) -> None:\n    while True:\n        await randsleep(caller=f\"Consumer {name}\")\n        i, t = await q.get()\n        now = time.perf_counter()\n        print(f\"Consumer {name} got element <{i}>\"\n              f\" in {now-t:0.5f} seconds.\")\n        q.task_done()\n\nasync def main(nprod: int, ncon: int):\n    q = asyncio.Queue()\n    producers = [asyncio.create_task(produce(n, q)) for n in range(nprod)]\n    consumers = [asyncio.create_task(consume(n, q)) for n in range(ncon)]\n    await asyncio.gather(*producers)\n    await q.join()  # Implicitly awaits consumers, too\n    for c in consumers:\n        c.cancel()\n\nif __name__ == \"__main__\":\n    import argparse\n    random.seed(444)\n    parser = argparse.ArgumentParser()\n    parser.add_argument(\"-p\", \"--nprod\", type=int, default=5)\n    parser.add_argument(\"-c\", \"--ncon\", type=int, default=10)\n    ns = parser.parse_args()\n    start = time.perf_counter()\n    asyncio.run(main(**ns.__dict__))\n    elapsed = time.perf_counter() - start\n    print(f\"Program completed in {elapsed:0.5f} seconds.\")\n

    The challenging part of this workflow is that there needs to be a signal to the consumers that production is done. Otherwise, await q.get() will hang indefinitely, because the queue will have been fully processed, but consumers won\u2019t have any idea that production is complete. The key is to await q.join(), which blocks until all items in the queue have been received and processed, and then to cancel the consumer tasks, which would otherwise hang up and wait endlessly for additional queue items to appear.

    The first few coroutines are helper functions that return a random string, a fractional-second performance counter, and a random integer. A producer puts anywhere from 1 to 10 items into the queue. Each item is a tuple of (i, t) where i is a random string and t is the time at which the producer attempts to put the tuple into the queue.

    When a consumer pulls an item out, it simply calculates the elapsed time that the item sat in the queue using the timestamp that the item was put in with.

    Here is a test run with two producers and five consumers:

    $ python3 asyncq.py -p 2 -c 5\nProducer 0 sleeping for 3 seconds.\nProducer 1 sleeping for 3 seconds.\nConsumer 0 sleeping for 4 seconds.\nConsumer 1 sleeping for 3 seconds.\nConsumer 2 sleeping for 3 seconds.\nConsumer 3 sleeping for 5 seconds.\nConsumer 4 sleeping for 4 seconds.\nProducer 0 added <377b1e8f82> to queue.\nProducer 0 sleeping for 5 seconds.\nProducer 1 added <413b8802f8> to queue.\nConsumer 1 got element <377b1e8f82> in 0.00013 seconds.\nConsumer 1 sleeping for 3 seconds.\nConsumer 2 got element <413b8802f8> in 0.00009 seconds.\nConsumer 2 sleeping for 4 seconds.\nProducer 0 added <06c055b3ab> to queue.\nProducer 0 sleeping for 1 seconds.\nConsumer 0 got element <06c055b3ab> in 0.00021 seconds.\nConsumer 0 sleeping for 4 seconds.\nProducer 0 added <17a8613276> to queue.\nConsumer 4 got element <17a8613276> in 0.00022 seconds.\nConsumer 4 sleeping for 5 seconds.\nProgram completed in 9.00954 seconds.\n

    In this case, the items process in fractions of a second. A delay can be due to two reasons:

    With regards to the second reason, luckily, it is perfectly normal to scale to hundreds or thousands of consumers. You should have no problem with python3 asyncq.py -p 5 -c 100. The point here is that, theoretically, you could have different users on different systems controlling the management of producers and consumers, with the queue serving as the central throughput.

    "}, {"location": "asyncio/#async-for-and-list-comprehensions", "title": "async for and list comprehensions", "text": "

    You can use async for to iterate over an asynchronous iterator. The purpose of an asynchronous iterator is for it to be able to call asynchronous code at each stage when it is iterated over.

    A natural extension of this concept is an asynchronous generator:

    >>> async def mygen(u: int = 10):\n...     \"\"\"Yield powers of 2.\"\"\"\n...     i = 0\n...     while i < u:\n...         yield 2 ** i\n...         i += 1\n...         await asyncio.sleep(0.1)\n

    You can also use asynchronous comprehension with async for:

    >>> async def main():\n...     # This does *not* introduce concurrent execution\n...     # It is meant to show syntax only\n...     g = [i async for i in mygen()]\n...     f = [j async for j in mygen() if not (j // 3 % 5)]\n...     return g, f\n...\n>>> g, f = asyncio.run(main())\n>>> g\n[1, 2, 4, 8, 16, 32, 64, 128, 256, 512]\n>>> f\n[1, 2, 16, 32, 256, 512]\n

    Asynchronous iterators and asynchronous generators are not designed to concurrently map some function over a sequence or iterator. They\u2019re merely designed to let the enclosing coroutine allow other tasks to take their turn. The async for and async with statements are only needed to the extent that using plain for or with would \u201cbreak\u201d the nature of await in the coroutine.

    "}, {"location": "asyncio/#the-event-loop-and-asynciorun", "title": "The Event Loop and asyncio.run()", "text": "

    You can think of an event loop as something like a while True loop that monitors coroutines, taking feedback on what\u2019s idle, and looking around for things that can be executed in the meantime. It is able to wake up an idle coroutine when whatever that coroutine is waiting on becomes available.

    Thus far, the entire management of the event loop has been implicitly handled by one function call:

    asyncio.run(main())\n

    asyncio.run() is responsible for getting the event loop, running tasks until they are marked as complete, and then closing the event loop.

    If you do need to interact with the event loop within a Python program, loop (obtained through loop = asyncio.get_event_loop()) is a good-old-fashioned Python object that supports introspection with loop.is_running() and loop.is_closed(). You can manipulate it if you need to get more fine-tuned control, such as in scheduling a callback by passing the loop as an argument.

    Some important points regarding the event loop are:

    "}, {"location": "asyncio/#creating-and-gathering-tasks", "title": "Creating and gathering tasks", "text": "

    You can use create_task() to schedule the execution of a coroutine object, followed by asyncio.run():

    >>> import asyncio\n\n>>> async def coro(seq) -> list:\n...     \"\"\"'IO' wait time is proportional to the max element.\"\"\"\n...     await asyncio.sleep(max(seq))\n...     return list(reversed(seq))\n...\n>>> async def main():\n...     # This is a bit redundant in the case of one task\n...     # We could use `await coro([3, 2, 1])` on its own\n...     t = asyncio.create_task(coro([3, 2, 1])) \n...     await t\n...     print(f't: type {type(t)}')\n...     print(f't done: {t.done()}')\n...\n>>> t = asyncio.run(main())\nt: type <class '_asyncio.Task'>\nt done: True\n

    There\u2019s a subtlety to this pattern: if you don\u2019t await t within main(), it may finish before main() itself signals that it is complete. Because asyncio.run(main()) calls loop.run_until_complete(main()), the event loop is only concerned (without await t present) that main() is done, not that the tasks that get created within main() are done, if this happens the loop\u2019s other tasks will be cancelled, possibly before they are completed. If you need to get a list of currently pending tasks, you can use asyncio.Task.all_tasks().

    Separately, there\u2019s asyncio.gather() which is meant to neatly put a collection of coroutines (futures) into a single future. As a result, it returns a single future object, and, if you await asyncio.gather() and specify multiple tasks or coroutines, you\u2019re waiting for all of them to be completed. (This somewhat parallels queue.join() from our earlier example.) The result of gather() will be a list of the results across the inputs:

    >>> import time\n>>> async def main():\n...     t = asyncio.create_task(coro([3, 2, 1]))\n...     t2 = asyncio.create_task(coro([10, 5, 0]))  # Python 3.7+\n...     print('Start:', time.strftime('%X'))\n...     a = await asyncio.gather(t, t2)\n...     print('End:', time.strftime('%X'))  # Should be 10 seconds\n...     print(f'Both tasks done: {all((t.done(), t2.done()))}')\n...     return a\n...\n>>> a = asyncio.run(main())\nStart: 16:20:11\nEnd: 16:20:21\nBoth tasks done: True\n>>> a\n[[1, 2, 3], [0, 5, 10]]\n

    You can loop over asyncio.as_completed() to get tasks as they are completed, in the order of completion. The function returns an iterator that yields tasks as they finish. Below, the result of coro([3, 2, 1]) will be available before coro([10, 5, 0]) is complete, which is not the case with gather():

    >>> async def main():\n...     t = asyncio.create_task(coro([3, 2, 1]))\n...     t2 = asyncio.create_task(coro([10, 5, 0]))\n...     print('Start:', time.strftime('%X'))\n...     for res in asyncio.as_completed((t, t2)):\n...         compl = await res\n...         print(f'res: {compl} completed at {time.strftime(\"%X\")}')\n...     print('End:', time.strftime('%X'))\n...     print(f'Both tasks done: {all((t.done(), t2.done()))}')\n...\n>>> a = asyncio.run(main())\nStart: 09:49:07\nres: [1, 2, 3] completed at 09:49:10\nres: [0, 5, 10] completed at 09:49:17\nEnd: 09:49:17\nBoth tasks done: True\n

    Lastly, you may also see asyncio.ensure_future(). You should rarely need it, because it\u2019s a lower-level plumbing API and largely replaced by create_task(), which was introduced later.

    "}, {"location": "asyncio/#when-and-why-is-async-io-the-right-choice", "title": "When and Why Is Async IO the Right Choice?", "text": "

    If you have multiple, fairly uniform CPU-bound tasks (a great example is a grid search in libraries such as scikit-learn or keras), multiprocessing should be an obvious choice.

    Simply putting async before every function is a bad idea if all of the functions use blocking calls. This can actually slow down your code.

    The contest between async IO and threading is a little bit more direct. Even in cases where threading seems easy to implement, it can still lead to infamous impossible-to-trace bugs due to race conditions and memory usage, among other things.

    Threading also tends to scale less elegantly than async IO, because threads are a system resource with a finite availability. Creating thousands of threads will fail on many machines. Creating thousands of async IO tasks is completely feasible.

    Async IO shines when you have multiple IO-bound tasks where the tasks would otherwise be dominated by blocking IO-bound wait time, such as:

    The biggest reason not to use it is that await only supports a specific set of objects that define a specific set of methods. If you want to do async read operations with a certain DBMS, you\u2019ll need to find not just a Python wrapper for that DBMS, but one that supports the async/await syntax. Coroutines that contain synchronous calls block other coroutines and tasks from running.

    "}, {"location": "asyncio/#async-io-it-is-but-which-one", "title": "Async IO It Is, but Which One?", "text": "

    asyncio certainly isn\u2019t the only async IO library out there. The most popular are:

    You might find that they get the same thing done in a way that\u2019s more intuitive for you as the user. Many of the package-agnostic concepts presented here should permeate to alternative async IO packages as well. But if you\u2019re building a moderately sized, straightforward program, just using asyncio is plenty sufficient and understandable, and lets you avoid adding yet another large dependency outside of Python\u2019s standard library.

    "}, {"location": "asyncio/#snippets", "title": "Snippets", "text": ""}, {"location": "asyncio/#write-on-file", "title": "Write on file", "text": "
    import aiofiles\nasync with aiofiles.open(file, \"a\") as f:\n    for p in res:\n        await f.write(f\"{url}\\t{p}\\n\")\n
    "}, {"location": "asyncio/#do-http-requests", "title": "Do http requests", "text": "

    Use the aiohttp library

    "}, {"location": "asyncio/#tips", "title": "Tips", "text": ""}, {"location": "asyncio/#limit-concurrency", "title": "Limit concurrency", "text": "

    Use asyncio.Semaphore.

    sem = asyncio.Semaphore(10)\nasync with sem:\n    # work with shared resource\n

    Note that this method is not thread-safe.

    "}, {"location": "asyncio/#testing", "title": "Testing", "text": "

    With the pytest-asyncio plugin you can test code that uses the asyncio library.

    Install it with pip install pytest-asyncio

    Specifically, pytest-asyncio provides support for coroutines as test functions. This allows users to await code inside their tests. For example, the following code is executed as a test item by pytest:

    @pytest.mark.asyncio\nasync def test_some_asyncio_code():\n    res = await library.do_something()\n    assert b\"expected result\" == res\n

    pytest-asyncio runs each test item in its own asyncio event loop. The loop can be accessed via the event_loop fixture, which is automatically requested by all async tests.

    async def test_provided_loop_is_running_loop(event_loop):\n    assert event_loop is asyncio.get_running_loop()\n

    You can think of event_loop as an autouse fixture for async tests.

    It has two discovery modes:

    To use the Auto mode you need to pass the --asyncio-mode=auto flag to pytest. If you use pyproject.toml you can set the next configuration:

    [tool.pytest.ini_options]\naddopts = \"--asyncio-mode=auto\"\n
    "}, {"location": "asyncio/#references", "title": "References", "text": ""}, {"location": "asyncio/#libraries-to-explore", "title": "Libraries to explore", "text": ""}, {"location": "authentik/", "title": "Authentik", "text": "

    Authentik is an open-source Identity Provider focused on flexibility and versatility.

    What I like:

    What I don't like:

    "}, {"location": "authentik/#installation", "title": "Installation", "text": "

    You can install it with Kubernetes or with docker-compose. I'm going to do the second.

    Download the latest docker-compose.yml from here. Place it in a directory of your choice.

    If this is a fresh authentik install run the following commands to generate a password:

    # You can also use openssl instead: `openssl rand -base64 36`\nsudo apt-get install -y pwgen\n# Because of a PostgreSQL limitation, only passwords up to 99 chars are supported\n# See https://www.postgresql.org/message-id/09512C4F-8CB9-4021-B455-EF4C4F0D55A0@amazon.com\necho \"PG_PASS=$(pwgen -s 40 1)\" >> .env\necho \"AUTHENTIK_SECRET_KEY=$(pwgen -s 50 1)\" >> .env\n

    It is also recommended to configure global email credentials. These are used by authentik to notify you about alerts and configuration issues. They can also be used by Email stages to send verification/recovery emails.

    Append this block to your .env file

    # SMTP Host Emails are sent to\nAUTHENTIK_EMAIL__HOST=localhost\nAUTHENTIK_EMAIL__PORT=25\n# Optionally authenticate (don't add quotation marks to your password)\nAUTHENTIK_EMAIL__USERNAME=\nAUTHENTIK_EMAIL__PASSWORD=\n# Use StartTLS\nAUTHENTIK_EMAIL__USE_TLS=false\n# Use SSL\nAUTHENTIK_EMAIL__USE_SSL=false\nAUTHENTIK_EMAIL__TIMEOUT=10\n# Email address authentik will send from, should have a correct @domain\nAUTHENTIK_EMAIL__FROM=authentik@localhost\n

    By default, authentik listens on port 9000 for HTTP and 9443 for HTTPS. To change this, you can set the following variables in .env:

    AUTHENTIK_PORT_HTTP=80\nAUTHENTIK_PORT_HTTPS=443\n

    You may need to tweak the volumes and the networks sections of the docker-compose.yml to your liking.

    Once everything is set you can run docker-compose up to test everything is working.

    In your browser, navigate to authentik\u2019s initial setup page https://auth.home.yourdomain.com/if/flow/initial-setup/

    Set the email and password for the default admin user, akadmin. You\u2019re now logged in.

    "}, {"location": "authentik/#configuration", "title": "Configuration", "text": ""}, {"location": "authentik/#terraform", "title": "Terraform", "text": "

    You can use terraform to configure authentik! <3.

    "}, {"location": "authentik/#configure-the-provider", "title": "Configure the provider", "text": "

    To configure the provider you need to specify the url and an Authentik API token, keeping in mind that whoever gets access to this information will have access and full permissions on your Authentik instance it's critical that you store this information well. We'll use sops to encrypt the token with GPG..

    First create an Authentik user under Admin interface/Directory/Users with the next attributes:

    Then create a token with name Terraform under Directory/Tokens & App passwords, copy it to your clipboard.

    Configure sops by defining the gpg keys in a .sops.yaml file at the top of your repository:

    ---\ncreation_rules:\n  - pgp: >-\n      2829BASDFHWEGWG23WDSLKGL323534J35LKWERQS,\n      2GEFDBW349YHEDOH2T0GE9RH0NEORIG342RFSLHH\n

    Then create the secrets file with the command sops secrets.enc.json somewhere in your terraform repository. For example:

    {\n  \"authentik_token\": \"paste the token here\"\n}\n
    terraform {\n  required_providers {\n    authentik = {\n      source = \"goauthentik/authentik\"\n      version = \"~> 2023.1.1\"\n    }\n    sops = {\n      source = \"carlpett/sops\"\n      version = \"~> 0.5\"\n    }\n  }\n}\n\nprovider \"authentik\" {\n  url   = \"https://oauth.your-domain.org\"\n  token = data.sops_file.secrets.data[\"authentik_token\"]\n}\n
    "}, {"location": "authentik/#configure-some-common-applications", "title": "Configure some common applications", "text": "

    You have some guides to connect some popular applications

    "}, {"location": "authentik/#gitea", "title": "Gitea", "text": "

    You can follow the Authentik Gitea docs or you can use the next terraform snippet:

    # ----------------\n# --    Data    --\n# ----------------\n\ndata \"authentik_flow\" \"default-authorization-flow\" {\n  slug = \"default-provider-authorization-implicit-consent\"\n}\n\n# -----------------------\n# --    Application    --\n# -----------------------\n\nresource \"authentik_application\" \"gitea\" {\n  name              = \"Gitea\"\n  slug              = \"gitea\"\n  protocol_provider = authentik_provider_oauth2.gitea.id\n  meta_icon = \"application-icons/gitea.svg\"\n  lifecycle {\n    ignore_changes = [\n      # The terraform provider is continuously changing the attribute even though it's set\n      meta_icon,\n    ]\n  }\n}\n\n# --------------------------\n# --    Oauth provider    --\n# --------------------------\n\nresource \"authentik_provider_oauth2\" \"gitea\" {\n  name               = \"Gitea\"\n  client_id = \"gitea\"\n  authorization_flow = data.authentik_flow.default-authorization-flow.id\n  property_mappings = [\n    authentik_scope_mapping.gitea.id,\n    data.authentik_scope_mapping.email.id,\n    data.authentik_scope_mapping.openid.id,\n    data.authentik_scope_mapping.profile.id,\n  ]\n  redirect_uris = [\n    \"https://git.your-domain.org/user/oauth2/authentik/callback\",\n  ]\n  signing_key = data.authentik_certificate_key_pair.default.id\n}\n\ndata \"authentik_certificate_key_pair\" \"default\" {\n  name = \"authentik Self-signed Certificate\"\n}\n\n# -------------------------\n# --    Scope mapping    --\n# -------------------------\n\nresource \"authentik_scope_mapping\" \"gitea\" {\n  name       = \"Gitea\"\n  scope_name = \"gitea\"\n  expression = <<EOF\ngitea_claims = {}\nif request.user.ak_groups.filter(name=\"Users\").exists():\n    gitea_claims[\"gitea\"]= \"user\"\nif request.user.ak_groups.filter(name=\"Admins\").exists():\n    gitea_claims[\"gitea\"]= \"admin\"\n\nreturn gitea_claims\nEOF\n}\n\ndata \"authentik_scope_mapping\" \"email\" {\n  managed = \"goauthentik.io/providers/oauth2/scope-email\"\n}\n\ndata \"authentik_scope_mapping\" \"openid\" {\n  managed = \"goauthentik.io/providers/oauth2/scope-openid\"\n}\n\ndata \"authentik_scope_mapping\" \"profile\" {\n  managed = \"goauthentik.io/providers/oauth2/scope-profile\"\n}\n\n# -------------------\n# --    Outputs    --\n# -------------------\n\noutput \"gitea_oauth_id\" {\n  value = authentik_provider_oauth2.gitea.client_id\n}\n\noutput \"gitea_oauth_secret\" {\n  value = authentik_provider_oauth2.gitea.client_secret\n}\n

    It assumes that:

    Gitea can be configured through terraform too. There is an official provider that doesn't work, there's a [fork that does though[(https://registry.terraform.io/providers/Lerentis/gitea/latest/docs). Sadly it doesn't yet support configuring Oauth Authentication sources. So you'll need to configure it manually.

    Be careful gitea_oauth2_app looks to be the right resource to do that, but instead it configures Gitea to be the Oauth provider, not a consumer.

    "}, {"location": "authentik/#configure-the-invitation-flow", "title": "Configure the invitation flow", "text": "

    Let's assume that we have two groups (Admins and Users) created under Directory/Groups and that we want to configure an invitation link for a user to be added directly on the Admins group.

    Authentik works by defining Stages and Flows. Stages are the steps you need to follow to complete a procedure, and a flow is the procedure itself.

    You create Stages by: * Going to the Admin interface * Going to Flows & Stages/Stages * Click on Create

    To be able to complete the invitation through link we need to define the next stages:

    Or use the next terraform snippet:

    resource \"authentik_stage_invitation\" \"default\" {\n  name                             = \"enrollment-invitation\"\n  continue_flow_without_invitation = false\n}\n

    Or use the next terraform snippet:

    resource \"authentik_stage_user_write\" \"admin_write\" {\n  name                     = \"enrollment-invitation-admin-write\"\n  create_users_as_inactive = true\n  create_users_group       = authentik_group.admins.id\n}\n

    Where authentik_group.admin is defined as:

    resource \"authentik_group\" \"admins\" {\n  name         = \"Admins\"\n  is_superuser = true\n  users = [\n    data.authentik_user.user_1.id,\n    data.authentik_user.user_2.id,\n  ]\n}\n\ndata \"authentik_user\" \"user_1\" {\n  username = \"user_1\"\n}\n\ndata \"authentik_user\" \"user_2\" {\n  username = \"user_2\"\n}\n

    Graphically you would need to:

    Or use the next terraform snippet:

    resource \"authentik_stage_email\" \"account_confirmation\" {\n  name                     = \"email-account-confirmation\"\n  activate_user_on_success = true\n  subject                  = \"Authentik Account Confirmation\"\n  template                 = \"email/account_confirmation.html\"\n  timeout                  = 10\n}\n

    Create the invitation Flow:

    Graphically you would need to:

    Or use the next terraform snippet:

    resource \"authentik_flow\" \"enrollment_admin\" {\n  name        = \"Enrollment invitation admin\"\n  title       = \"Enrollment invitation admin\"\n  slug        = \"enrollment-invitation-admin\"\n  designation = \"enrollment\"\n}\n

    We need to define how the flow is going to behave by adding the different the stage bindings:

    Graphically you would need to:

    Or use the next terraform snippet:

    resource \"authentik_flow_stage_binding\" \"invitation_creation\" {\n  target = authentik_flow.enrollment_admin.uuid\n  stage  = authentik_stage_invitation.default.id\n  order  = 10\n}\n

    Graphically you would need to:

    Or use the next terraform snippet:

    resource \"authentik_stage_prompt\" \"user_data\" {\n  name = \"enrollment-user-data-prompt\"\n  fields = [ \n      authentik_stage_prompt_field.username.id,\n      authentik_stage_prompt_field.name.id,\n      authentik_stage_prompt_field.email.id,\n      authentik_stage_prompt_field.password.id,\n      authentik_stage_prompt_field.password_repeat.id,\n  ]\n}\n\nresource \"authentik_stage_prompt_field\" \"username\" {\n  field_key = \"username\"\n  label     = \"Username\"\n  type      = \"text\"\n  order = 200\n  placeholder = <<EOT\ntry:\n    return user.username\nexcept:\n    return ''\nEOT\n  placeholder_expression = true\n  required = true\n}\n\nresource \"authentik_stage_prompt_field\" \"name\" {\n  field_key = \"name\"\n  label     = \"Name\"\n  type      = \"text\"\n  order = 201\n  placeholder = <<EOT\ntry:\n    return user.name\nexcept:\n    return ''\nEOT\n  placeholder_expression = true\n  required = true\n}\n\nresource \"authentik_stage_prompt_field\" \"email\" {\n  field_key = \"email\"\n  label     = \"Email\"\n  type      = \"email\"\n  order = 202\n  placeholder = <<EOT\ntry:\n    return user.email\nexcept:\n    return ''\nEOT\n  placeholder_expression = true\n  required = true\n}\n\nresource \"authentik_stage_prompt_field\" \"password\" {\n  field_key = \"password\"\n  label     = \"Password\"\n  type      = \"password\"\n  order = 300\n  placeholder = \"Password\"\n  placeholder_expression = false\n  required = true\n}\n\nresource \"authentik_stage_prompt_field\" \"password_repeat\" {\n  field_key = \"password_repeat\"\n  label     = \"Password (repeat)\"\n  type      = \"password\"\n  order = 301\n  placeholder = \"Password (repeat)\"\n  placeholder_expression = false\n  required = true\n}\n

    We had to redefine all the authentik_stage_prompt_field because the terraform provider doesn't yet support the data resource of the authentik_stage_prompt_field

    Graphically you would need to:

    Or use the next terraform snippet:

    resource \"authentik_flow_stage_binding\" \"invitation_user_write\" {\n  target = authentik_flow.enrollment_admin.uuid\n  stage  = authentik_stage_user_write.admin_write.id\n  order  = 30\n}\n

    Graphically you would need to:

    Or use the next terraform snippet:

    resource \"authentik_flow_stage_binding\" \"invitation_account_confirmation\" {\n  target = authentik_flow.enrollment_admin.uuid\n  stage  = authentik_stage_email.account_confirmation.id\n  order  = 40\n}\n

    Graphically you would need to:

    Or use the next terraform snippet:

    resource \"authentik_flow_stage_binding\" \"invitation_login\" {\n  target = authentik_flow.enrollment_admin.uuid\n  stage  = data.authentik_stage.default_source_enrollment_login.id\n  order  = 50\n}\n
    "}, {"location": "authentik/#configure-password-recovery", "title": "Configure password recovery", "text": "

    Recovery of password is not enabled by default, to configure it you need to create two new stages:

    data \"authentik_source\" \"built_in\" {\n  managed = \"goauthentik.io/sources/inbuilt\"\n}\n\nresource \"authentik_stage_identification\" \"recovery\" {\n  name           = \"recovery-authentication-identification\"\n  user_fields    = [\"username\", \"email\"]\n  sources = [data.authentik_source.built_in.uuid]\n  case_insensitive_matching = true\n}\n
    resource \"authentik_stage_email\" \"recovery\" {\n  name                     = \"recovery-email\"\n  activate_user_on_success = true\n  subject                  = \"Password Recovery\"\n  template                 = \"email/password_reset.html\"\n  timeout                  = 10\n}\n
    data \"authentik_stage\" \"default_password_change_prompt\" {\n  name = \"default-password-change-prompt\"\n}\n\ndata \"authentik_stage\" \"default_password_change_write\" {\n  name = \"default-password-change-write\"\n}\n

    Then we need to create the recovery flow and bind all the stages:

    resource \"authentik_flow\" \"password_recovery\" {\n  name        = \"Password Recovery\"\n  title       = \"Password Recovery\"\n  slug        = \"password-recovery\"\n  designation = \"recovery\"\n}\n\nresource \"authentik_flow_stage_binding\" \"recovery_identification\" {\n  target = authentik_flow.password_recovery.uuid\n  stage  = authentik_stage_identification.recovery.id\n  order  = 0\n}\n\nresource \"authentik_flow_stage_binding\" \"recovery_email\" {\n  target = authentik_flow.password_recovery.uuid\n  stage  = authentik_stage_email.recovery.id\n  order  = 10\n}\n\nresource \"authentik_flow_stage_binding\" \"recovery_password_change\" {\n  target = authentik_flow.password_recovery.uuid\n  stage  = data.authentik_stage.default_password_change_prompt.id\n  order  = 20\n}\n\nresource \"authentik_flow_stage_binding\" \"recovery_password_write\" {\n  target = authentik_flow.password_recovery.uuid\n  stage  = data.authentik_stage.default_password_change_write.id\n  order  = 30\n}\n

    Finally we need to enable it in the site's authentication flow. To be able to do change the default flow we'd need to do two manual steps, so to have all the code in terraform we will create a new tenancy for our site and a new authentication flow.

    Starting with the authentication flow we need to create the Flow, stages and stage bindings.

    # -----------\n# -- Flows --\n# -----------\n\nresource \"authentik_flow\" \"authentication\" {\n  name        = \"Welcome to Authentik!\"\n  title        = \"Welcome to Authentik!\"\n  slug        = \"custom-authentication-flow\"\n  designation = \"authentication\"\n  authentication = \"require_unauthenticated\"\n  compatibility_mode = false\n}\n\n# ------------\n# -- Stages --\n# ------------\n\nresource \"authentik_stage_identification\" \"authentication\" {\n  name           = \"custom-authentication-identification\"\n  user_fields    = [\"username\", \"email\"]\n  password_stage = data.authentik_stage.default_authentication_password.id\n  case_insensitive_matching = true\n  recovery_flow = authentik_flow.password_recovery.uuid\n}\n\ndata \"authentik_stage\" \"default_authentication_mfa_validation\" {\n  name = \"default-authentication-mfa-validation\"\n}\n\ndata \"authentik_stage\" \"default_authentication_login\" {\n  name = \"default-authentication-login\"\n}\n\ndata \"authentik_stage\" \"default_authentication_password\" {\n  name = \"default-authentication-password\"\n}\n\n# -------------------\n# -- Stage binding --\n# -------------------\n\nresource \"authentik_flow_stage_binding\" \"login_identification\" {\n  target = authentik_flow.authentication.uuid\n  stage  = authentik_stage_identification.authentication.id\n  order  = 10\n}\n\nresource \"authentik_flow_stage_binding\" \"login_mfa\" {\n  target = authentik_flow.authentication.uuid\n  stage  = data.authentik_stage.default_authentication_mfa_validation.id\n  order  = 20\n}\n\nresource \"authentik_flow_stage_binding\" \"login_login\" {\n  target = authentik_flow.authentication.uuid\n  stage  = data.authentik_stage.default_authentication_login.id\n  order  = 30\n}\n

    Now we can bind it to the new tenant for our site:

    # ------------\n# -- Tenant --\n# ------------\n\nresource \"authentik_tenant\" \"default\" {\n  domain         = \"your-domain.org\"\n  default        = false\n  branding_title = \"Authentik\"\n  branding_logo = \"/static/dist/assets/icons/icon_left_brand.svg\"\n  branding_favicon = \"/static/dist/assets/icons/icon.png\"\n  flow_authentication = authentik_flow.authentication.uuid\n  # We need to define id instead of uuid until \n  # https://github.com/goauthentik/terraform-provider-authentik/issues/305\n  # is fixed.\n  flow_invalidation = data.authentik_flow.default_invalidation_flow.id\n  flow_user_settings = data.authentik_flow.default_user_settings_flow.id\n  flow_recovery = authentik_flow.password_recovery.uuid\n}\n\n# -----------\n# -- Flows --\n# -----------\n\ndata \"authentik_flow\" \"default_invalidation_flow\" {\n  slug = \"default-invalidation-flow\"\n}\n\ndata \"authentik_flow\" \"default_user_settings_flow\" {\n  slug = \"default-user-settings-flow\"\n}\n
    "}, {"location": "authentik/#hide-and-application-from-a-user", "title": "Hide and application from a user", "text": "

    Application access can be configured using (Policy) Bindings. Click on an application in the applications list, and select the Policy / Group / User Bindings tab. There you can bind users/groups/policies to grant them access. When nothing is bound, everyone has access. You can use this to grant access to one or multiple users/groups, or dynamically give access using policies.

    With terraform you can use authentik_policy_binding, for example:

    resource \"authentik_policy_binding\" \"admin\" {\n  target = authentik_application.gitea.uuid\n  group  = authentik_group.admins.id\n  order  = 0\n}\n
    "}, {"location": "authentik/#protect-applications-that-dont-have-authentication", "title": "Protect applications that don't have authentication", "text": "

    Some applications don't have authentication, for example prometheus. You can use Authentik in front of such applications to add the authentication and authorization layer.

    Authentik can be used as a (very) simple reverse proxy by using its Provider feature with the regular \"Proxy\" setting. This let's you wrap authentication around a sub-domain / app where it normally wouldn't have authentication (or not the type of auth that you would specifically want) and then have Authentik handle the proxy forwarding and Auth.

    In this mode, there is no domain level nor 'integrated' authentication into your desired app; Authentik becomes both your reverse proxy and auth for this one particular app or (sub) domain. This mode does not forward authentication nor let you log in into any app. It's just acts like an authentication wrapper.

    It's best to use a normal reverse proxy out front of Authentik. This adds a second layer of routing to deal with but Authentik is not NGINX or a reverse proxy system, so it does not have that many configuration options.

    We'll use the following fake domains in this example:

    The steps are:

    # ---------------\n# -- Variables --\n# ---------------\n\nvariable \"prometheus_url\" {\n  type        = string\n  description = \"The url to access the service.\"\n}\n\n# ----------\n# -- Data --\n# ----------\n\ndata \"authentik_flow\" \"default-authorization-flow\" {\n  slug = \"default-provider-authorization-implicit-consent\"\n}\n\n# --------------------\n# --    Provider    --\n# --------------------\n\nresource \"authentik_provider_proxy\" \"prometheus\" {\n  name               = \"Prometheus\"\n  internal_host      = \"http://prometheus:9090\"\n  external_host      = var.prometheus_url\n  authorization_flow = data.authentik_flow.default-authorization-flow.id\n  internal_host_ssl_validation = false\n}\n
    variable \"prometheus_icon\" {\n  type        = string\n  description = \"The icon shown in the application\"\n  default     = \"/application-icons/prometheus.svg\"\n}\n\n# -----------------------\n# --    Application    --\n# -----------------------\n\nresource \"authentik_application\" \"prometheus\" {\n  name              = \"Prometheus\"\n  slug              = \"prometheus\"\n  meta_icon         = var.prometheus_icon\n  protocol_provider = authentik_provider_proxy.prometheus.id\n  lifecycle {\n    ignore_changes = [\n      # The terraform provider is continuously changing the attribute even though it's set\n      meta_icon,\n    ]\n  }\n}\n
    # ----------------\n# --- Outposts ---\n# ----------------\n\nresource \"authentik_outpost\" \"default\" {\n  name = \"authentik Embedded Outpost\"\n  service_connection = authentik_service_connection_docker.local.id \n  protocol_providers = [\n    authentik_provider_proxy.prometheus.id\n  ]\n}\n\n# ----------------------------\n# --- Outpost integrations ---\n# ----------------------------\n\nresource \"authentik_service_connection_docker\" \"local\" {\n  name  = \"Local Docker connection\"\n  local = true\n}\n
    "}, {"location": "authentik/#use-blueprints", "title": "Use blueprints", "text": "

    WARNING: Use the terraform provider instead!!!

    Blueprints offer a new way to template, automate and distribute authentik configuration. Blueprints can be used to automatically configure instances, manage config as code without any external tools, and to distribute application configs.

    Blueprints are yaml files, whose format is described further in File structure and uses YAML tags to configure the objects. It can be complicated when you first look at it, reading this example may help.

    Blueprints can be applied in one of two ways:

    The authentik container by default looks for blueprints in /blueprints. Underneath this directory, there are a couple default subdirectories:

    Any additional .yaml file in /blueprints will be discovered and automatically instantiated, depending on their labels.

    To disable existing blueprints, an empty file can be mounted over the existing blueprint.

    File-based blueprints are automatically removed once they become unavailable, however none of the objects created by those blueprints are affected by this.

    "}, {"location": "authentik/#export-blueprints", "title": "Export blueprints", "text": "

    Exports from either method will contain a (potentially) long list of objects, all with hardcoded primary keys and no ability for templating/instantiation. This is because currently, authentik does not check which primary keys are used where. It is assumed that for most exports, there'll be some manual changes done regardless, to filter out unwanted objects, adjust properties, etc. That's why it may be better to use the flow export for the resources you've created rather than the global export.

    "}, {"location": "authentik/#global-export", "title": "Global export", "text": "

    To migrate existing configurations to blueprints, run ak export_blueprint within any authentik Worker container. This will output a blueprint for most currently created objects. Some objects will not be exported as they might have dependencies on other things.

    Exported blueprints don't use any of the YAML Tags, they just contain a list of entries as they are in the database.

    Note that fields which are write-only (for example, OAuth Provider's Secret Key) will not be added to the blueprint, as the serialisation logic from the API is used for blueprints.

    Additionally, default values will be skipped and not added to the blueprint.

    "}, {"location": "authentik/#flow-export", "title": "Flow export", "text": "

    Instead of exporting everything from a single instance, there's also the option to export a single flow with it's attached stages, policies and other objects.

    This export can be triggered via the API or the Web UI by clicking the download button in the flow list.

    "}, {"location": "authentik/#monitorization", "title": "Monitorization", "text": "

    I've skimmed through the prometheus metrics exposed at :9300/metrics in the core and they aren't that useful :(

    "}, {"location": "authentik/#troubleshooting", "title": "Troubleshooting", "text": ""}, {"location": "authentik/#i-cant-log-in-to-authentik", "title": "I can't log in to authentik", "text": "

    In case you can't login anymore, perhaps due to an incorrectly configured stage or a failed flow import, you can create a recovery key.

    To create the key, run the following command:

    docker run --it authentik bash\nak create_recovery_key 1 akadmin\n

    This will output a link, that can be used to instantly gain access to authentik as the user specified above. The link is valid for amount of years specified above, in this case, 1 year.

    "}, {"location": "authentik/#references", "title": "References", "text": ""}, {"location": "aws_savings_plan/", "title": "AWS Savings plan", "text": "

    Saving plans offer a flexible pricing model that provides savings on AWS usage. You can save up to 72 percent on your AWS compute workloads.

    !!! note \"Please don't make Jeff Bezos even richer, try to pay as less money to AWS as you can.\"

    Savings Plans provide savings beyond On-Demand rates in exchange for a commitment of using a specified amount of compute power (measured per hour) for a one or three year period.

    When you sign up for Savings Plans, the prices you'll pay for usage stays the same through the plan term. You can pay for your commitment using All Upfront, Partial upfront, or No upfront payment options.

    Plan types:

    "}, {"location": "aws_savings_plan/#understanding-how-savings-plans-apply-to-your-aws-usage", "title": "Understanding how Savings Plans apply to your AWS usage", "text": "

    If you have active Savings Plans, they apply automatically to your eligible AWS usage to reduce your bill.

    Savings Plans apply to your usage after the Amazon EC2 Reserved Instances (RI) are applied. Then EC2 Instance Savings Plans are applied before Compute Savings Plans because Compute Savings Plans have broader applicability.

    They calculate your potential savings percentages of each combination of eligible usage. This percentage compares the Savings Plans rates with your current On-Demand rates. Your Savings Plans are applied to your highest savings percentage first. If there are multiple usages with equal savings percentages, Savings Plans are applied to the first usage with the lowest Savings Plans rate. Savings Plans continue to apply until there are no more remaining usages, or your commitment is exhausted. Any remaining usage is charged at the On-Demand rates.

    "}, {"location": "aws_savings_plan/#savings-plan-example", "title": "Savings plan example", "text": "

    In this example, you have the following usage in a single hour:

    Pricing example:

    Type On-Demand rate Compute Savings Plans rate CSP Savings percentage EC2 Instance Savings Plans rate EC2IS percentage r5.4xlarge $1.00 $0.70 30% $0.60 40% m5.24xlarge $10.00 $8.20 18% $7.80 22%

    They've included other products in the example but I've removed them for the sake of simplicity

    "}, {"location": "aws_savings_plan/#scenario-1-savings-plan-apply-to-all-usage", "title": "Scenario 1: Savings Plan apply to all usage", "text": "

    You purchase a one-year, partial upfront Compute Savings Plan with a $50.00/hour commitment.

    Your Savings Plan covers all of your usage because multiplying each of your usages by the equivalent Compute Savings Plans is $47.13. This is still less than the $50.00/hour commitment.

    Without Savings Plans, you would be charged at On-Demand rates in the amount of $59.10.

    "}, {"location": "aws_savings_plan/#scenario-2-savings-plans-apply-to-some-usage", "title": "Scenario 2: Savings Plans apply to some usage", "text": "

    You purchase a one-year, partial upfront Compute Savings Plan with a $2.00/hour commitment.

    In any hour, your Savings Plans apply to your usage starting with the highest discount percentage (30 percent).

    Your $2.00/hour commitment is used to cover approximately 2.9 units of this usage. The remaining 1.1 units are charged at On-Demand rates, resulting in $1.14 of On-Demand charges for r5.

    The rest of your usage are also charged at On-Demand rates, resulting in $55.10 of On-Demand charges. The total On-Demand charges for this usage are $56.24.

    "}, {"location": "aws_savings_plan/#scenario-3-savings-plans-and-ec2-reserved-instances-apply-to-the-usage", "title": "Scenario 3: Savings Plans and EC2 reserved instances apply to the usage", "text": "

    You purchase a one-year, partial upfront Compute Savings Plan with an $18.20/hour commitment. You have two EC2 Reserved Instances (RI) for r5.4xlarge Linux shared tenancy in us-east-1.

    First, the Reserve Instances covers two of the r5.4xlarge instances. Then, the Savings Plans rate is applied to the remaining r5.4xlarge and the rest of the usage, which exhausts the hourly commitment of $18.20.

    "}, {"location": "aws_savings_plan/#scenario-4-multiple-savings-plans-apply-to-the-usage", "title": "Scenario 4: Multiple Savings Plans apply to the usage", "text": "

    You purchase a one-year, partial upfront EC2 Instance Family Savings Plan for the r5 family in us-east-1 with a $3.00/hour commitment. You also have a one-year, partial upfront Compute Savings Plan with a $16.80/hour commitment.

    Your EC2 Instance Family Savings Plan (r5, us-east-1) covers all of the r5.4xlarge usage because multiplying the usage by the EC2 Instance Family Savings Plan rate is $2.40. This is less than the $3.00/hour commitment.

    Next, the Compute Savings Plan is applied to rest of the resource usage, if it doesn't cover the whole expense, then On demand rates will apply.

    "}, {"location": "aws_savings_plan/#monitoring-the-savings-plan", "title": "Monitoring the savings plan", "text": "

    Monitoring is an important part of your Savings Plans usage. Understanding the Savings Plan that you own, how they are applying to your usage, and what usage is being covered are important parts of optimizing your costs with Savings Plans. You can monitor your usage in multiple forms.

    "}, {"location": "aws_savings_plan/#doing-your-savings-plan", "title": "Doing your savings plan", "text": "

    Go to the AWS savings plan simulator and check the different instances you were evaluating.

    "}, {"location": "aws_snippets/", "title": "AWS Snippets", "text": ""}, {"location": "aws_snippets/#invalidate-a-cloudfront-distribution", "title": "Invalidate a cloudfront distribution", "text": "
    aws cloudfront create-invalidation --paths \"/pages/about\" --distribution-id my-distribution-id\n
    "}, {"location": "aws_snippets/#get-ec2-metadata-from-within-the-instance", "title": "Get EC2 metadata from within the instance", "text": "

    The quickest way to fetch or retrieve EC2 instance metadata from within a running EC2 instance is to log in and run the command:

    Fetch metadata from IPv4:

    curl -s http://169.254.169.254/latest/dynamic/instance-identity/document\n

    You can also download the ec2-metadata tool to get the info:

    # Download the ec2-metadata script\nwget http://s3.amazonaws.com/ec2metadata/ec2-metadata\n\n# Modify the permission to execute the bash script\nchmod +x ec2-metadata\n\n./ec2-metadata --all\n
    "}, {"location": "aws_snippets/#find-if-external-ip-belongs-to-you", "title": "Find if external IP belongs to you", "text": "

    You can list the network interfaces that match the IP you're searching for

    aws ec2 describe-network-interfaces --filters Name=association.public-ip,Values=\"{{ your_ip_address}}\"\n
    "}, {"location": "aws_waf/", "title": "AWS WAF", "text": "

    AWS WAF is a web application firewall that helps protect your web applications or APIs against common web exploits and bots that may affect availability, compromise security, or consume excessive resources. AWS WAF gives you control over how traffic reaches your applications by enabling you to create security rules that control bot traffic and block common attack patterns, such as SQL injection or cross-site scripting. You can also customize rules that filter out specific traffic patterns.

    "}, {"location": "aws_waf/#extracting-information", "title": "Extracting information", "text": "

    You can configure the WAF to write it's logs into S3, Kinesis or a Cloudwatch log group. S3 saves the data in small compressed files which are difficult to analyze, Kinesis makes sense if you post-process the data on a log system such as graylog, the last one allows you to use the WAF's builtin cloudwatch log insights which has the next interesting reports: . * Top 100 Ip addresses * Top 100 countries * Top 100 hosts * Top 100 terminating rules . Nevertheless, it still lacks some needed reports to analyze the traffic. But it's quite easy to build them yourself in Cloudwatch Log Insights. If you have time I'd always suggest to avoid using proprietary AWS tools, but sadly it's the quickest way to get results.

    "}, {"location": "aws_waf/#creating-log-insights-queries", "title": "Creating Log Insights queries", "text": "

    Inside the Cloudwatch site, on the left menu you'll see the Logs tab, and under it Log Insights. There you can write the query you want to run. Once it returns the expected result, you can save it. Saved queries can be seen on the right menu, under Queries.

    If you later change the query, you'll see a blue dot beside the query you last run. The query will remain changed until you click on Actions and then Reset.

    "}, {"location": "aws_waf/#useful-queries", "title": "Useful Queries", "text": ""}, {"location": "aws_waf/#top-ips", "title": "Top IPs", "text": "

    Is a directory to save the queries to analyze a count of requests aggregated by ips.

    "}, {"location": "aws_waf/#top-ips-query", "title": "Top IPs query", "text": "
    fields httpRequest.clientIp\n| stats count(*) as requestCount by httpRequest.clientIp\n| sort requestCount desc\n| limit 100\n
    "}, {"location": "aws_waf/#top-ips-by-uri", "title": "Top IPs by uri", "text": "
    fields httpRequest.clientIp\n| filter httpRequest.uri like \"/\"\n| stats count(*) as requestCount by httpRequest.clientIp\n| sort requestCount desc\n| limit 100\n
    "}, {"location": "aws_waf/#top-uris", "title": "Top URIs", "text": "

    Is a directory to save the queries to analyze a count of requests aggregated by uris.

    "}, {"location": "aws_waf/#top-uris-query", "title": "Top URIs query", "text": "

    This report shows all the uris that are allowed to pass the WAF.

    fields httpRequest.uri\n| filter action like \"ALLOW\"\n| stats count(*) as requestCount by httpRequest.uri\n| sort requestCount desc\n| limit 100\n
    "}, {"location": "aws_waf/#top-uris-of-a-termination-rule", "title": "Top URIs of a termination rule", "text": "
    fields httpRequest.uri\n| filter terminatingRuleId like \"AWS-AWSManagedRulesUnixRuleSet\"\n| stats count(*) as requestCount by httpRequest.uri\n| sort requestCount desc\n| limit 100\n
    "}, {"location": "aws_waf/#top-uris-of-an-ip", "title": "Top URIs of an IP", "text": "
    fields httpRequest.uri\n| filter @message like \"6.132.241.132\"\n| stats count(*) as requestCount by httpRequest.uri\n| sort requestCount desc\n| limit 100\n
    "}, {"location": "aws_waf/#top-uris-of-a-cloudfront-id", "title": "Top URIs of a Cloudfront ID", "text": "
    fields httpRequest.uri\n| filter httpSourceId like \"CLOUDFRONT_ID\"\n| stats count(*) as requestCount by httpRequest.uri\n| sort requestCount desc\n| limit 100\n
    "}, {"location": "aws_waf/#waf-top-terminating-rules", "title": "WAF Top terminating rules", "text": "

    Report that shows the top rules that are blocking the content.

    fields terminatingRuleId\n| filter terminatingRuleId not like \"Default_Action\"\n| stats count(*) as requestCount by terminatingRuleId\n| sort requestCount desc\n| limit 100\n
    "}, {"location": "aws_waf/#top-blocks-by-cloudfront-id", "title": "Top blocks by Cloudfront ID", "text": "
    fields httpSourceId\n| filter terminatingRuleId not like \"Default_Action\"\n| stats count(*) as requestCount by httpSourceId\n| sort requestCount desc\n| limit 100\n
    "}, {"location": "aws_waf/#top-allows-by-cloudfront-id", "title": "Top allows by Cloudfront ID", "text": "
    fields httpSourceId\n| filter terminatingRuleId like \"Default_Action\"\n| stats count(*) as requestCount by httpSourceId\n| sort requestCount desc\n| limit 100\n
    "}, {"location": "aws_waf/#waf-top-countries", "title": "WAF Top countries", "text": "
    fields httpRequest.country\n| stats count(*) as requestCount by httpRequest.country\n| sort requestCount desc\n| limit 100\n
    "}, {"location": "aws_waf/#requests-by", "title": "Requests by", "text": "

    Is a directory to save the queries to show the requests filtered by a criteria.

    "}, {"location": "aws_waf/#requests-by-ip", "title": "Requests by IP", "text": "
    fields @timestamp, httpRequest.uri, httpRequest.args, httpSourceId\n| sort @timestamp desc\n| filter @message like \"6.132.241.132\"\n| limit 100\n
    "}, {"location": "aws_waf/#requests-by-termination-rule", "title": "Requests by termination rule", "text": "
    fields @timestamp, httpRequest.uri, httpRequest.args, httpSourceId\n| sort @timestamp desc\n| filter terminatingRuleId like \"AWS-AWSManagedRulesUnixRuleSet\"\n| limit 100\n
    "}, {"location": "aws_waf/#requests-by-uri", "title": "Requests by URI", "text": "
    fields @timestamp, httpRequest.uri, httpRequest.args, httpSourceId\n| sort @timestamp desc\n| filter httpRequest.uri like \"wp-json\"\n| limit 100\n
    "}, {"location": "aws_waf/#waf-top-args-of-an-uri", "title": "WAF Top Args of an URI", "text": "
    fields httpRequest.args\n| filter httpRequest.uri like \"/\"\n| stats count(*) as requestCount by httpRequest.args\n| sort requestCount desc\n| limit 100\n
    "}, {"location": "aws_waf/#analysis-workflow", "title": "Analysis workflow", "text": "

    To analyze the WAF insights you can:

    "}, {"location": "aws_waf/#analyze-the-traffic-of-the-top-ips", "title": "Analyze the traffic of the top IPS", "text": "

    For IP in the WAF Top IPs report, do:

    "}, {"location": "aws_waf/#analyze-the-top-uris", "title": "Analyze the top uris", "text": "

    For uri in the WAF Top URIs report, do:

    "}, {"location": "aws_waf/#analyze-the-terminating-rules", "title": "Analyze the terminating rules", "text": "

    For terminating rule in the WAF Top terminating rules report, do:

    After some time you can see which rules are not being triggered and remove them. With the requests by termination rule you can see which requests are being blocked and try to block it in another rule set and merge both.

    "}, {"location": "aws_waf/#mark-ip-as-problematic", "title": "Mark IP as problematic", "text": "

    To process an problematic IP:

    "}, {"location": "aws_waf/#references", "title": "References", "text": ""}, {"location": "bash_snippets/", "title": "Bash Snippets", "text": ""}, {"location": "bash_snippets/#loop-through-a-list-of-files-found-by-find", "title": "Loop through a list of files found by find", "text": "

    For simple loops use the find -exec syntax:

    # execute `process` once for each file\nfind . -name '*.txt' -exec process {} \\;\n

    For more complex loops use a while read construct:

    find . -name \"*.txt\" -print0 | while read -r -d $'\\0' file\ndo\n    \u2026code using \"$file\"\ndone\n

    The loop will execute while the find command is executing. Plus, this command will work even if a file name is returned with whitespace in it. And, you won't overflow your command line buffer.

    The -print0 will use the NULL as a file separator instead of a newline and the -d $'\\0' will use NULL as the separator while reading.

    "}, {"location": "bash_snippets/#how-not-to-do-it", "title": "How not to do it", "text": "

    If you try to run the next snippet:

    # Don't do this\nfor file in $(find . -name \"*.txt\")\ndo\n    \u2026code using \"$file\"\ndone\n

    You'll get the next shellcheck warning:

    SC2044: For loops over find output are fragile. Use find -exec or a while read loop.\n

    You should not do this because:

    Three reasons:

    "}, {"location": "bash_snippets/#remove-the-lock-screen-in-ubuntu", "title": "Remove the lock screen in ubuntu", "text": "

    Create the /usr/share/glib-2.0/schemas/90_ubuntu-settings.gschema.override file with the next content:

    [org.gnome.desktop.screensaver]\nlock-enabled = false\n[org.gnome.settings-daemon.plugins.power]\nidle-dim = false\n

    Then reload the schemas with:

    sudo glib-compile-schemas /usr/share/glib-2.0/schemas/\n
    "}, {"location": "bash_snippets/#how-to-deal-with-hostcontextswitching-alertmanager-alert", "title": "How to deal with HostContextSwitching alertmanager alert", "text": "

    A context switch is described as the kernel suspending execution of one process on the CPU and resuming execution of some other process that had previously been suspended. A context switch is required for every interrupt and every task that the scheduler picks.

    Context switching can be due to multitasking, Interrupt handling , user & kernel mode switching. The interrupt rate will naturally go high, if there is higher network traffic, or higher disk traffic. Also it is dependent on the application which every now and then invoking system calls.

    If the cores/CPU's are not sufficient to handle load of threads created by application will also result in context switching.

    It is not a cause of concern until performance breaks down. This is expected that CPU will do context switching. One shouldn't verify these data at first place since there are many statistical data which should be analyzed prior to looking into kernel activities. Verify the CPU, memory and network usage during this time.

    You can see which process is causing issue with the next command:

    # pidstat -w 3 10   > /tmp/pidstat.out\n\n10:15:24 AM     UID     PID     cswch/s         nvcswch/s       Command \n10:15:27 AM     0       1       162656.7        16656.7         systemd\n10:15:27 AM     0       9       165451.04       15451.04        ksoftirqd/0\n10:15:27 AM     0       10      158628.87       15828.87        rcu_sched\n10:15:27 AM     0       11      156147.47       15647.47        migration/0\n10:15:27 AM     0       17      150135.71       15035.71        ksoftirqd/1\n10:15:27 AM     0       23      129769.61       12979.61        ksoftirqd/2\n10:15:27 AM     0       29      2238.38         238.38          ksoftirqd/3\n10:15:27 AM     0       43      1753            753             khugepaged\n10:15:27 AM     0       443     1659            165             usb-storage\n10:15:27 AM     0       456     1956.12         156.12          i915/signal:0\n10:15:27 AM     0       465     29550           29550           kworker/3:1H-xfs-log/dm-3\n10:15:27 AM     0       490     164700          14700           kworker/0:1H-kblockd\n10:15:27 AM     0       506     163741.24       16741.24        kworker/1:1H-xfs-log/dm-3\n10:15:27 AM     0       594     154742          154742          dmcrypt_write/2\n10:15:27 AM     0       629     162021.65       16021.65        kworker/2:1H-kblockd\n10:15:27 AM     0       715     147852.48       14852.48        xfsaild/dm-1\n10:15:27 AM     0       886     150706.86       15706.86        irq/131-iwlwifi\n10:15:27 AM     0       966     135597.92       13597.92        xfsaild/dm-3\n10:15:27 AM     81      1037    2325.25         225.25          dbus-daemon\n10:15:27 AM     998     1052    118755.1        11755.1         polkitd\n10:15:27 AM     70      1056    158248.51       15848.51        avahi-daemon\n10:15:27 AM     0       1061    133512.12       455.12          rngd\n10:15:27 AM     0       1110    156230          16230           cupsd\n10:15:27 AM     0       1192    152298.02       1598.02         sssd_nss\n10:15:27 AM     0       1247    166132.99       16632.99        systemd-logind\n10:15:27 AM     0       1265    165311.34       16511.34        cups-browsed\n10:15:27 AM     0       1408    10556.57        1556.57         wpa_supplicant\n10:15:27 AM     0       1687    3835            3835            splunkd\n10:15:27 AM     42      1773    3728            3728            Xorg\n10:15:27 AM     42      1996    3266.67         266.67          gsd-color\n10:15:27 AM     0       3166    32036.36        3036.36         sssd_kcm\n10:15:27 AM     119349  3194    151763.64       11763.64        dbus-daemon\n10:15:27 AM     119349  3199    158306          18306           Xorg\n10:15:27 AM     119349  3242    15.28           5.8             gnome-shell\n\n# pidstat -wt 3 10  > /tmp/pidstat-t.out\n\nLinux 4.18.0-80.11.2.el8_0.x86_64 (hostname)    09/08/2020  _x86_64_    (4 CPU)\n\n10:15:15 AM   UID      TGID       TID   cswch/s   nvcswch/s  Command\n10:15:19 AM     0         1         -   152656.7   16656.7   systemd\n10:15:19 AM     0         -         1   152656.7   16656.7   |__systemd\n10:15:19 AM     0         9         -   165451.04  15451.04  ksoftirqd/0\n10:15:19 AM     0         -         9   165451.04  15451.04  |__ksoftirqd/0\n10:15:19 AM     0        10         -   158628.87  15828.87  rcu_sched\n10:15:19 AM     0         -        10   158628.87  15828.87  |__rcu_sched\n10:15:19 AM     0        23         -   129769.61  12979.61  ksoftirqd/2\n10:15:19 AM     0         -        23   129769.61  12979.33  |__ksoftirqd/2\n10:15:19 AM     0        29         -   32424.5    2445      ksoftirqd/3\n10:15:19 AM     0         -        29   32424.5    2445      |__ksoftirqd/3\n10:15:19 AM     0        43         -   334        34        khugepaged\n10:15:19 AM     0         -        43   334        34        |__khugepaged\n10:15:19 AM     0       443         -   11465      566       usb-storage\n10:15:19 AM     0         -       443   6433       93        |__usb-storage\n10:15:19 AM     0       456         -   15.41      0.00      i915/signal:0\n10:15:19 AM     0         -       456   15.41      0.00      |__i915/signal:0\n10:15:19 AM     0       715         -   19.34      0.00      xfsaild/dm-1\n10:15:19 AM     0         -       715   19.34      0.00      |__xfsaild/dm-1\n10:15:19 AM     0       886         -   23.28      0.00      irq/131-iwlwifi\n10:15:19 AM     0         -       886   23.28      0.00      |__irq/131-iwlwifi\n10:15:19 AM     0       966         -   19.67      0.00      xfsaild/dm-3\n10:15:19 AM     0         -       966   19.67      0.00      |__xfsaild/dm-3\n10:15:19 AM    81      1037         -   6.89       0.33      dbus-daemon\n10:15:19 AM    81         -      1037   6.89       0.33      |__dbus-daemon\n10:15:19 AM     0      1038         -   11567.31   4436      NetworkManager\n10:15:19 AM     0         -      1038   1.31       0.00      |__NetworkManager\n10:15:19 AM     0         -      1088   0.33       0.00      |__gmain\n10:15:19 AM     0         -      1094   1340.66    0.00      |__gdbus\n10:15:19 AM   998      1052         -   118755.1   11755.1   polkitd\n10:15:19 AM   998         -      1052   32420.66   25545     |__polkitd\n10:15:19 AM   998         -      1132   0.66       0.00      |__gdbus\n

    Then with help of PID which is causing issue, one can get all system calls details: Raw

    # strace -c -f -p <pid of process/thread>\n

    Let this command run for a few minutes while the load/context switch rates are high. It is safe to run this on a production system so you could run it on a good system as well to provide a comparative baseline. Through strace, one can debug & troubleshoot the issue, by looking at system calls the process has made.

    "}, {"location": "bash_snippets/#redirect-stderr-of-all-subsequent-commands-of-a-script-to-a-file", "title": "Redirect stderr of all subsequent commands of a script to a file", "text": "
    {\n    somecommand \n    somecommand2\n    somecommand3\n} 2>&1 | tee -a $DEBUGLOG\n
    "}, {"location": "bash_snippets/#get-the-root-path-of-a-git-repository", "title": "Get the root path of a git repository", "text": "
    git rev-parse --show-toplevel\n
    "}, {"location": "bash_snippets/#get-epoch-gmt-time", "title": "Get epoch gmt time", "text": "
    date -u '+%s'\n
    "}, {"location": "bash_snippets/#check-the-length-of-an-array-with-jq", "title": "Check the length of an array with jq", "text": "
    echo '[{\"username\":\"user1\"},{\"username\":\"user2\"}]' | jq '. | length'\n
    "}, {"location": "bash_snippets/#exit-the-script-if-there-is-an-error", "title": "Exit the script if there is an error", "text": "
    set -eu\n
    "}, {"location": "bash_snippets/#prompt-the-user-for-data", "title": "Prompt the user for data", "text": "
    read -p \"Ask whatever\" choice\n
    "}, {"location": "bash_snippets/#parse-csv-with-bash", "title": "Parse csv with bash", "text": ""}, {"location": "bash_snippets/#do-the-remainder-or-modulus-of-a-number", "title": "Do the remainder or modulus of a number", "text": "
    expr 5 % 3\n
    "}, {"location": "bash_snippets/#update-a-json-file-with-jq", "title": "Update a json file with jq", "text": "

    Save the next snippet to a file, for example jqr and add it to your $PATH.

    #!/bin/zsh\n\nquery=\"$1\"\nfile=$2\n\ntemp_file=\"$(mktemp)\"\n\n# Update the content\njq \"$query\" $file > \"$temp_file\"\n\n# Check if the file has changed\ncmp -s \"$file\" \"$temp_file\"\nif [[ $? -eq 0 ]] ; then\n  /bin/rm \"$temp_file\"\nelse\n  /bin/mv \"$temp_file\" \"$file\"\nfi\n

    Imagine you have the next json file:

    {\n  \"property\": true,\n  \"other_property\": \"value\"\n}\n

    Then you can run:

    jqr '.property = false' status.json\n

    And then you'll have:

    {\n  \"property\": false,\n  \"other_property\": \"value\"\n}\n
    "}, {"location": "beancount/", "title": "Beancount", "text": "

    Beancount is a Python double entry accounting command line tool similar to ledger.

    "}, {"location": "beancount/#installation", "title": "Installation", "text": "
    pip3 install beancount\n
    "}, {"location": "beancount/#tools", "title": "Tools", "text": "

    beancount is the core component, it's a declarative language. It parses a text file, and produces reports from the resulting data structures.

    "}, {"location": "beancount/#bean-check", "title": "bean-check", "text": "

    bean-check is the program you use to verify that your input syntax and transactions work correctly. All it does is load your input file and run the various plugins you configured in it, plus some extra validation checks.

    bean-check /path/to/file.beancount\n

    If there are no errors, there should be no output, it should exit quietly.

    "}, {"location": "beancount/#bean-report", "title": "bean-report", "text": "

    This is the main tool used to extract specialized reports to the console in text or one of the various other formats.

    For a graphic exploration of your data, use the fava web application instead.

    bean-report /path/to/file.beancount {{ report_name }}\n

    There are many reports available, to get a full list run bean-report --help-reports

    Report names sometimes may accept arguments, if they do so use :

    bean-report /path/to/file.beancount balances:Vanguard\n
    "}, {"location": "beancount/#to-get-the-balances", "title": "To get the balances", "text": "
    bean-report {{ path/to/file.beancount }} balances | treeify\n
    "}, {"location": "beancount/#to-get-the-journal", "title": "To get the journal", "text": "
    bean-report {{ path/to/file.beancount }} journal\n
    "}, {"location": "beancount/#to-get-the-holdings", "title": "To get the holdings", "text": "

    To get the aggregations for the total list of holdings

    bean-report {{ path/to/file.beancount }} holdings\n
    "}, {"location": "beancount/#to-get-the-accounts", "title": "To get the accounts", "text": "
    bean-report {{ path/to/file.beancount }} accounts\n
    "}, {"location": "beancount/#bean-query", "title": "bean-query", "text": "

    bean-query is a command-line tool that acts like a client to that in-memory database in which you can type queries in a variant of SQL. It has it's own document

    bean-query /path/to/file.beancount\n
    "}, {"location": "beancount/#bean-web", "title": "bean-web", "text": "

    Deprecated use fava instead

    bean-web serves all the reports on a web server that runs on your computer

    bean-web /path/to/file.beancount\n

    It will serve on localhost:8080

    "}, {"location": "beancount/#bean-doctor", "title": "bean-doctor", "text": "

    This is a debugging tool used to perform various diagnostics and run debugging commands, and to help provide information for reporting bugs.

    "}, {"location": "beancount/#bean-format", "title": "bean-format", "text": "

    Pure text processing tool will reformat Beancount input to right-align all the numbers at the same, minimal column.

    "}, {"location": "beancount/#bean-example", "title": "bean-example", "text": "

    Generates an example Beancount input file.

    "}, {"location": "beancount/#bean-identify", "title": "bean-identify", "text": "

    Given a messy list of downloaded files automatically identify which of your configured importers is able to handle them and print them out. This is to be used for debugging and figuring out if your configuration is properly associating a suitable importer for each of the files you downloaded.

    "}, {"location": "beancount/#bean-extract", "title": "bean-extract", "text": "

    Extracts transactions and statement date from each file, if at all possible. This produces some Beancount input text to be moved to your input file.

    bean-extract {{ path/to/config.config }} {{ path/to/source/files }}\n

    The tool calls methods on importer objects. You must provide a list of such importers; this list is the configuration for the importing process.

    For each file found, each of the importers is called to assert whether it can or cannot handle that file. If it deems that it can, methods can be called to produce a list of transactions extract a date, or produce a cleaned up filename for the downloaded file.

    The configuration should be a python3 module in which you instantiate the importers and assign the list to the module-level \"CONFIG\" variable

    #!/usr/bin/env python3\nfrom myimporters.bank import acmebank\nfrom myimporters.bank import chase\n\u2026\nCONFIG = [\nacmebank.Importer(),\nchase.Importer(),\n\u2026\n]\n
    "}, {"location": "beancount/#writing-an-importer", "title": "Writing an importer", "text": "

    Each of the importers must comply with a particular protocol and implement at least some of its methods. The full detail of the protocol is in the source of importer.py

    \"\"\"Importer protocol.\n\nAll importers must comply with this interface and implement at least some of its\nmethods. A configuration consists in a simple list of such importer instances.\nThe importer processes run through the importers, calling some of its methods in\norder to identify, extract and file the downloaded files.\n\nEach of the methods accept a cache.FileMemo object which has a 'name' attribute\nwith the filename to process, but which also provides a place to cache\nconversions. Use its convert() method whenever possible to avoid carrying out\nthe same conversion multiple times. See beancount.ingest.cache for more details.\n\nSynopsis:\n\n name(): Return a unique identifier for the importer instance.\n identify(): Return true if the identifier is able to process the file.\n extract(): Extract directives from a file's contents and return of list of entries.\n file_account(): Return an account name associated with the given file for this importer.\n file_date(): Return a date associated with the downloaded file (e.g., the statement date).\n file_name(): Return a cleaned up filename for storage (optional).\n\nJust to be clear: Although this importer will not raise NotImplementedError\nexceptions (it returns default values for each method), you NEED to derive from\nit in order to do anything meaningful. Simply instantiating this importer will\nnot match not provide any useful information. It just defines the protocol for\nall importers.\n\"\"\"\n__copyright__ = \"Copyright (C) 2016  Martin Blais\"\n__license__ = \"GNU GPLv2\"\n\nfrom beancount.core import flags\n\n\nclass ImporterProtocol:\n    \"Interface that all source importers need to comply with.\"\n\n    # A flag to use on new transaction. Override this flag in derived classes if\n    # you prefer to create your imported transactions with a different flag.\n    FLAG = flags.FLAG_OKAY\n\n    def name(self):\n        \"\"\"Return a unique id/name for this importer.\n\n        Returns:\n          A string which uniquely identifies this importer.\n        \"\"\"\n        cls = self.__class__\n        return '{}.{}'.format(cls.__module__, cls.__name__)\n\n    __str__ = name\n\n    def identify(self, file):\n        \"\"\"Return true if this importer matches the given file.\n\n        Args:\n          file: A cache.FileMemo instance.\n        Returns:\n          A boolean, true if this importer can handle this file.\n        \"\"\"\n\n    def extract(self, file):\n        \"\"\"Extract transactions from a file.\n\n        Args:\n          file: A cache.FileMemo instance.\n        Returns:\n          A list of new, imported directives (usually mostly Transactions)\n          extracted from the file.\n        \"\"\"\n\n    def file_account(self, file):\n        \"\"\"Return an account associated with the given file.\n\n        Note: If you don't implement this method you won't be able to move the\n        files into its preservation hierarchy; the bean-file command won't work.\n\n        Also, normally the returned account is not a function of the input\n        file--just of the importer--but it is provided anyhow.\n\n        Args:\n          file: A cache.FileMemo instance.\n        Returns:\n          The name of the account that corresponds to this importer.\n        \"\"\"\n\n    def file_name(self, file):\n        \"\"\"A filter that optionally renames a file before filing.\n\n        This is used to make tidy filenames for filed/stored document files. The\n        default implementation just returns the same filename. Note that a\n        simple RELATIVE filename must be returned, not an absolute filename.\n\n        Args:\n          file: A cache.FileMemo instance.\n        Returns:\n          The tidied up, new filename to store it as.\n        \"\"\"\n\n    def file_date(self, file):\n        \"\"\"Attempt to obtain a date that corresponds to the given file.\n\n        Args:\n          file: A cache.FileMemo instance.\n        Returns:\n          A date object, if successful, or None if a date could not be extracted.\n          (If no date is returned, the file creation time is used. This is the\n          default.)\n

    A summary of the methods you need to, or may want to implement:

    from beancount.ingest import importer\n\nclass Importer(importer.ImporterProtocol):\n\n  def identify(self, file):\n  \u2026\n  # Override other methods\u2026\n

    Some importer examples:

    "}, {"location": "beancount/#bean-file", "title": "bean-file", "text": ""}, {"location": "beancount/#basic-concepts", "title": "Basic concepts", "text": ""}, {"location": "beancount/#beancount-transaction", "title": "Beancount transaction", "text": "
    2014-05-23 * \"CAFE MOGADOR NEW YO\" \"Dinner with Caroline\"\n  Liabilities:US:BofA:CreditCard -98.32 USD\n  Expenses:Restaurant\n
    "}, {"location": "beancount/#beancount-operators", "title": "Beancount Operators", "text": ""}, {"location": "beancount/#open", "title": "Open", "text": "

    All accounts need to be declared open in order to accept amounts posted to them.

    YYYY-MM-DD open {{ account_name }} [{{ ConstrainCurrency }}]\n
    "}, {"location": "beancount/#close", "title": "Close", "text": "
    YYYY-MM-DD close {{ account_name }}\n

    It's useful to insert a balance of 0 units just before closing an account, just to make sure its contents are empty as you close it.

    "}, {"location": "beancount/#commodity", "title": "Commodity", "text": "

    It can be used to declare currencies, financial instruments, commodities... It's optional

    YYYY-MM-DD commodity {{ currency_name }}\n
    "}, {"location": "beancount/#transactions", "title": "Transactions", "text": "
    YYYY-MM-DD txn \"[{{ payee }}]\"  \"{{ Comment }}\"\n  {{ Account1 }} {{ value}}\n  [{{ Accountn-1 }} {{ value }}]\n  {{ Accountn }}\n

    Payee is a string that represents an external entity that is involved in the transaction. Payees are sometimes useful on transactions that post amounts to Expense accounts, whereby the account accumulates a category of expenses from multiple business

    As transactions is the most common, you can substitute txn for a flag, by default : * *: Completed transaction, known amounts, \"this looks correct\" * !: Incomplete transaction, needs confirmation or revision, \"this looks incorrect\"

    You can also attach flags to the postings themselves, if you want to flag one of the transaction's legs in particular:

    2014-05-05 * \"Transfer from Savings account\"\n  Assets:MyBank:Checking     -400.00 USD\n  ! Assets:MyBank:Savings\n

    This is useful in the intermediate stage of de-duping transactions

    "}, {"location": "beancount/#tags-vs-payee", "title": "Tags vs Payee", "text": "

    You can tag your transactions with #{{tag_name}}, so you can later filter or generate reports based on that tag. Therefore the Payee could be used as whom or who pays and the tag for the context. For example, for a trip I could use the tag #34C3

    To mark a series of transactions with tags use the following syntax 

    pushtag #berlin-trip-2014\n\n2014-04-23 * \"Flight to Berlin\"\n  Expenses:Flights -1230.27 USD\n  Liabilities:CreditCard\n\n...\n\npoptag #berlin-trip-2014\n

    "}, {"location": "beancount/#links", "title": "Links", "text": "

    Transactions can also be linked together. You may think of the link as a special kind of tag that can be used to group together a set of financially related transactions over time.

    2014-02-05 * \"Invoice for January\" ^invoice-acme-studios-jan14\n  Income:Clients:ACMEStudios   -8450.00 USD\n  Assets:AccountsReceivable\n\n...\n\n2014-02-20 * \"Check deposit - payment from ACME\" ^invoice-acme-studios-jan14\n  Assets:BofA:Checking         8450.00 USD\n  Assets:AccountsReceivable\n
    "}, {"location": "beancount/#balance", "title": "Balance", "text": "

    A balance assertion is a way for you to input your statement balance into the flow of transactions. It tells Beancount to verify that the number of units of a particular commodity in some account should equal some expected value at some point in time.

    If no error is reported, you should have some confidence that the list of transactions that precedes it in this account is highly likely to be correct. This is useful in practice because in many cases some transactions can get imported separately from the accounts of each of their postings.

    As all other non-transaction directives, it applies at the beginning of it's date. Just imagine that the balance checks occurs right after midnight on that day.

    YYYY-MM-DD balance {{ account_name }} {{ amount }}\n
    "}, {"location": "beancount/#pad", "title": "Pad", "text": "

    A padding directive automatically inserts a transaction that will make the subsequent balance assertion succeed, if it is needed. It inserts the difference needed to fulfill that balance assertion.

    Being subsequent in date order, not in the order of the declarations in the file.

    YYYY-MM-DD pad {{ account_name }} {{ account_name_to_pad }}\n

    The first account is the account to credit the automatically calculated amount to. This is the account that should have a balance assertion following it. The second leg is the source where the funds will come from, and this is almost always some Equity account.

    1990-05-17 open Assets:Cash EUR\n1990-05-17 pad Assets:Cash Equity:Opening-Balances\n2017-12-26 balance Assets:Cash 250 EUR\n

    You could also insert pad entries between balance assertions so as to fix un registered transactions

    "}, {"location": "beancount/#notes", "title": "Notes", "text": "

    A note directive is simply used to attach a dated comment to the journal of a particular account.

    this can be useful to record facts and claims associated with a financial event.

    YYYY-MM-DD note {{ account_name }} {{ comment }}\n
    "}, {"location": "beancount/#document", "title": "Document", "text": "

    A Document directive can be used to attach an external file to the journal of an account.

    The filename gets rendered as a browser link in the journals of the web interface for the corresponding account and you should be able to click on it to view the contents of the file itself.

    YYYY-MM-DD {{ account_name }} {{ path/to/document }}\n
    "}, {"location": "beancount/#includes", "title": "Includes", "text": "

    This allows you to split up large input files into multiple files.

    include {{ path/to/file.beancount }}\n

    The path could be relative or absolute.

    "}, {"location": "beancount/#library-usage", "title": "Library usage", "text": "

    Beancount can also be used as a Python library.

    There are some articles in the documentation where you can start seeing how to use it: scripting plugins , external contributions and the api reference. Although I found it more pleasant to read the source code itself as it's really well documented (both by docstrings and type hints).

    "}, {"location": "beancount/#references", "title": "References", "text": ""}, {"location": "beautifulsoup/", "title": "BeautifulSoup", "text": "

    BeautifulSoup is a Python library for pulling data out of HTML and XML files. It works with your favorite parser to provide idiomatic ways of navigating, searching, and modifying the parse tree.

    import requests\nfrom bs4 import BeautifulSoup\n\nrequest = requests.get('{{ url }}')\nsoup = BeautifulSoup(request.text, \"html.parser\")\n

    Here are some simple ways to navigate that data structure:

    soup.title\n# <title>The Dormouse's story</title>\n\nsoup.title.name\n# u'title'\n\nsoup.title.string\n# u'The Dormouse's story'\n\nsoup.title.parent.name\n# u'head'\n\nsoup.p\n# <p class=\"title\"><b>The Dormouse's story</b></p>\n\nsoup.p['class']\n# u'title'\n\nsoup.a\n# <a class=\"sister\" href=\"http://example.com/elsie\" id=\"link1\">Elsie</a>\n\nsoup.find_all('a')\n# [<a class=\"sister\" href=\"http://example.com/elsie\" id=\"link1\">Elsie</a>,\n#  <a class=\"sister\" href=\"http://example.com/lacie\" id=\"link2\">Lacie</a>,\n#  <a class=\"sister\" href=\"http://example.com/tillie\" id=\"link3\">Tillie</a>]\n\nsoup.find(id=\"link3\")\n# <a class=\"sister\" href=\"http://example.com/tillie\" id=\"link3\">Tillie</a>\n
    "}, {"location": "beautifulsoup/#installation", "title": "Installation", "text": "
    pip install beautifulsoup4\n

    The default parser html.parser doesn't work with HTML5, so you'll probably need to use the html5lib parser, it's not included by default, so you might need to install it as well

    pip install html5lib\n
    "}, {"location": "beautifulsoup/#usage", "title": "Usage", "text": ""}, {"location": "beautifulsoup/#kinds-of-objects", "title": "Kinds of objects", "text": "

    Beautiful Soup transforms a complex HTML document into a complex tree of Python objects. But you\u2019ll only ever have to deal with about four kinds of objects: Tag, NavigableString, BeautifulSoup, and Comment.

    "}, {"location": "beautifulsoup/#tag", "title": "Tag", "text": "

    A Tag object corresponds to an XML or HTML tag in the original document:

    soup = BeautifulSoup('<b class=\"boldest\">Extremely bold</b>')\ntag = soup.b\ntype(tag)\n# <class 'bs4.element.Tag'>\n

    The most important features of a tag are its name and attributes.

    "}, {"location": "beautifulsoup/#name", "title": "Name", "text": "

    Every tag has a name, accessible as .name:

    tag.name\n# u'b'\n

    If you change a tag\u2019s name, the change will be reflected in any HTML markup generated by Beautiful Soup:.

    tag.name = \"blockquote\"\ntag\n# <blockquote class=\"boldest\">Extremely bold</blockquote>\n
    "}, {"location": "beautifulsoup/#attributes", "title": "Attributes", "text": "

    A tag may have any number of attributes. The tag <b id=\"boldest\"> has an attribute id whose value is boldest. You can access a tag\u2019s attributes by treating the tag like a dictionary:

    tag['id']\n# u'boldest'\n

    You can access that dictionary directly as .attrs:

    tag.attrs\n# {u'id': 'boldest'}\n

    You can add, remove, and modify a tag\u2019s attributes. Again, this is done by treating the tag as a dictionary:

    tag['id'] = 'verybold'\ntag['another-attribute'] = 1\ntag\n# <b another-attribute=\"1\" id=\"verybold\"></b>\n\ndel tag['id']\ndel tag['another-attribute']\ntag\n# <b></b>\n\ntag['id']\n# KeyError: 'id'\nprint(tag.get('id'))\n# None\n
    "}, {"location": "beautifulsoup/#multi-valued-attributes", "title": "Multi-valued attributes", "text": "

    HTML 4 defines a few attributes that can have multiple values. HTML 5 removes a couple of them, but defines a few more. The most common multi-valued attribute is class (that is, a tag can have more than one CSS class). Others include rel, rev, accept-charset, headers, and accesskey. Beautiful Soup presents the value(s) of a multi-valued attribute as a list:

    css_soup = BeautifulSoup('<p class=\"body\"></p>')\ncss_soup.p['class']\n# [\"body\"]\n\ncss_soup = BeautifulSoup('<p class=\"body strikeout\"></p>')\ncss_soup.p['class']\n# [\"body\", \"strikeout\"]\n

    If an attribute looks like it has more than one value, but it\u2019s not a multi-valued attribute as defined by any version of the HTML standard, Beautiful Soup will leave the attribute alone:

    id_soup = BeautifulSoup('<p id=\"my id\"></p>')\nid_soup.p['id']\n# 'my id'\n

    When you turn a tag back into a string, multiple attribute values are consolidated:

    rel_soup = BeautifulSoup('<p>Back to the <a rel=\"index\">homepage</a></p>')\nrel_soup.a['rel']\n# ['index']\nrel_soup.a['rel'] = ['index', 'contents']\nprint(rel_soup.p)\n# <p>Back to the <a rel=\"index contents\">homepage</a></p>\n

    If you parse a document as XML, there are no multi-valued attributes:

    "}, {"location": "beautifulsoup/#navigablestring", "title": "NavigableString", "text": "

    A string corresponds to a bit of text within a tag. Beautiful Soup uses the NavigableString class to contain these bits of text:

    tag.string\n# u'Extremely bold'\ntype(tag.string)\n# <class 'bs4.element.NavigableString'>\n

    A NavigableString is just like a Python Unicode string, except that it also supports some of the features described in Navigating the tree and Searching the tree. You can convert a NavigableString to a Unicode string with unicode():

    unicode_string = unicode(tag.string)\nunicode_string\n# u'Extremely bold'\ntype(unicode_string)\n# <type 'unicode'>\n

    You can\u2019t edit a string in place, but you can replace one string with another, using replace_with():

    tag.string.replace_with(\"No longer bold\")\ntag\n# <blockquote>No longer bold</blockquote>\n
    "}, {"location": "beautifulsoup/#beautifulsoup", "title": "BeautifulSoup", "text": "

    The BeautifulSoup object represents the parsed document as a whole. For most purposes, you can treat it as a Tag object. This means it supports most of the methods described in Navigating the tree and Searching the tree.

    "}, {"location": "beautifulsoup/#navigating-the-tree", "title": "Navigating the tree", "text": ""}, {"location": "beautifulsoup/#going-down", "title": "Going down", "text": "

    Tags may contain strings and other tags. These elements are the tag\u2019s children. Beautiful Soup provides a lot of different attributes for navigating and iterating over a tag\u2019s children.

    Note that Beautiful Soup strings don\u2019t support any of these attributes, because a string can\u2019t have children.

    "}, {"location": "beautifulsoup/#navigating-using-tag-names", "title": "Navigating using tag names", "text": "

    The simplest way to navigate the parse tree is to say the name of the tag you want. If you want the <head> tag, just say soup.head:

    soup.head\n# <head><title>The Dormouse's story</title></head>\n\nsoup.title\n# <title>The Dormouse's story</title>\n

    You can do use this trick again and again to zoom in on a certain part of the parse tree. This code gets the first <b> tag beneath the <body> tag:

    soup.body.b\n# <b>The Dormouse's story</b>\n

    Using a tag name as an attribute will give you only the first tag by that name:

    soup.a\n# <a class=\"sister\" href=\"http://example.com/elsie\" id=\"link1\">Elsie</a>\n

    If you need to get all the <a> tags, or anything more complicated than the first tag with a certain name, you\u2019ll need to use one of the methods described in Searching the tree, such as find_all():

    soup.find_all('a')\n# [<a class=\"sister\" href=\"http://example.com/elsie\" id=\"link1\">Elsie</a>,\n#  <a class=\"sister\" href=\"http://example.com/lacie\" id=\"link2\">Lacie</a>,\n#  <a class=\"sister\" href=\"http://example.com/tillie\" id=\"link3\">Tillie</a>]\n
    "}, {"location": "beautifulsoup/#contents-and-children", "title": ".contents and .children", "text": "

    A tag\u2019s children are available in a list called .contents:

    head_tag = soup.head\nhead_tag\n# <head><title>The Dormouse's story</title></head>\n\nhead_tag.contents\n[<title>The Dormouse's story</title>]\n\ntitle_tag = head_tag.contents[0]\ntitle_tag\n# <title>The Dormouse's story</title>\ntitle_tag.contents\n# [u'The Dormouse's story']\n

    Instead of getting them as a list, you can iterate over a tag\u2019s children using the .children generator:

    for child in title_tag.children:\n    print(child)\n# The Dormouse's story\n
    "}, {"location": "beautifulsoup/#descendants", "title": ".descendants", "text": "

    The .contents and .children attributes only consider a tag\u2019s direct children. For instance, the <head> tag has a single direct child\u2013the <title> tag:

    head_tag.contents\n# [<title>The Dormouse's story</title>]\n

    But the <title> tag itself has a child: the string The Dormouse\u2019s story. There\u2019s a sense in which that string is also a child of the <head> tag. The .descendants attribute lets you iterate over all of a tag\u2019s children, recursively: its direct children, the children of its direct children, and so on:.

    for child in head_tag.descendants:\n    print(child)\n# <title>The Dormouse's story</title>\n# The Dormouse's story\n
    "}, {"location": "beautifulsoup/#string", "title": ".string", "text": "

    If a tag has only one child, and that child is a NavigableString, the child is made available as .string:

    title_tag.string\n# u'The Dormouse's story'\n

    If a tag\u2019s only child is another tag, and that tag has a .string, then the parent tag is considered to have the same .string as its child:

    head_tag.contents\n# [<title>The Dormouse's story</title>]\n\nhead_tag.string\n# u'The Dormouse's story'\n

    If a tag contains more than one thing, then it\u2019s not clear what .string should refer to, so .string is defined to be None:

    print(soup.html.string)\n# None\n
    "}, {"location": "beautifulsoup/#strings-and-stripped_strings", "title": ".strings and .stripped_strings", "text": "

    If there\u2019s more than one thing inside a tag, you can still look at just the strings. Use the .strings generator:

    for string in soup.strings:\n    print(repr(string))\n# u\"The Dormouse's story\"\n# u'\\n\\n'\n# u\"The Dormouse's story\"\n# u'\\n\\n'\n

    These strings tend to have a lot of extra whitespace, which you can remove by using the .stripped_strings generator instead:

    for string in soup.stripped_strings:\n    print(repr(string))\n# u\"The Dormouse's story\"\n# u\"The Dormouse's story\"\n# u'Once upon a time there were three little sisters; and their names were'\n# u'Elsie'\n
    "}, {"location": "beautifulsoup/#going-up", "title": "Going up", "text": "

    Continuing the \u201cfamily tree\u201d analogy, every tag and every string has a parent: the tag that contains it.

    "}, {"location": "beautifulsoup/#parent", "title": ".parent", "text": "

    You can access an element\u2019s parent with the .parent attribute.

    title_tag = soup.title\ntitle_tag\n# <title>The Dormouse's story</title>\ntitle_tag.parent\n# <head><title>The Dormouse's story</title></head>\n
    "}, {"location": "beautifulsoup/#parents", "title": ".parents", "text": "

    You can iterate over all of an element\u2019s parents with .parents.

    "}, {"location": "beautifulsoup/#going-sideways", "title": "Going sideways", "text": "

    When a document is pretty-printed, siblings show up at the same indentation level. You can also use this relationship in the code you write.

    "}, {"location": "beautifulsoup/#next_sibling-and-previous_sibling", "title": ".next_sibling and .previous_sibling", "text": "

    You can use .next_sibling and .previous_sibling to navigate between page elements that are on the same level of the parse tree:.

    sibling_soup.b.next_sibling\n# <c>text2</c>\n\nsibling_soup.c.previous_sibling\n# <b>text1</b>\n

    The <b> tag has a .next_sibling, but no .previous_sibling, because there\u2019s nothing before the <b> tag on the same level of the tree. For the same reason, the <c> tag has a .previous_sibling but no .next_sibling:

    print(sibling_soup.b.previous_sibling)\n# None\nprint(sibling_soup.c.next_sibling)\n# None\n

    In real documents, the .next_sibling or .previous_sibling of a tag will usually be a string containing whitespace.

    <a href=\"http://example.com/elsie\" class=\"sister\" id=\"link1\">Elsie</a>\n<a href=\"http://example.com/lacie\" class=\"sister\" id=\"link2\">Lacie</a>\n<a href=\"http://example.com/tillie\" class=\"sister\" id=\"link3\">Tillie</a>\n

    You might think that the .next_sibling of the first <a> tag would be the second <a> tag. But actually, it\u2019s a string: the comma and newline that separate the first <a> tag from the second:

    link = soup.a\nlink\n# <a class=\"sister\" href=\"http://example.com/elsie\" id=\"link1\">Elsie</a>\n\nlink.next_sibling\n# u',\\n'\n

    The second <a> tag is actually the .next_sibling of the comma:

    link.next_sibling.next_sibling\n# <a class=\"sister\" href=\"http://example.com/lacie\" id=\"link2\">Lacie</a>\n
    "}, {"location": "beautifulsoup/#next_siblings-and-previous_siblings", "title": ".next_siblings and .previous_siblings", "text": "

    You can iterate over a tag\u2019s siblings with .next_siblings or .previous_siblings:

    for sibling in soup.a.next_siblings:\n    print(repr(sibling))\n# u',\\n'\n# <a class=\"sister\" href=\"http://example.com/lacie\" id=\"link2\">Lacie</a>\n# u' and\\n'\n# <a class=\"sister\" href=\"http://example.com/tillie\" id=\"link3\">Tillie</a>\n# u'; and they lived at the bottom of a well.'\n# None\n\nfor sibling in soup.find(id=\"link3\").previous_siblings:\n    print(repr(sibling))\n# ' and\\n'\n# <a class=\"sister\" href=\"http://example.com/lacie\" id=\"link2\">Lacie</a>\n# u',\\n'\n# <a class=\"sister\" href=\"http://example.com/elsie\" id=\"link1\">Elsie</a>\n# u'Once upon a time there were three little sisters; and their names were\\n'\n# None\n
    "}, {"location": "beautifulsoup/#searching-the-tree", "title": "Searching the tree", "text": "

    By passing in a filter to an argument like find_all(), you can zoom in on the parts of the document you\u2019re interested in.

    "}, {"location": "beautifulsoup/#kinds-of-filters", "title": "Kinds of filters", "text": ""}, {"location": "beautifulsoup/#a-string", "title": "A string", "text": "

    The simplest filter is a string. Pass a string to a search method and Beautiful Soup will perform a match against that exact string. This code finds all the <b> tags in the document:

    soup.find_all('b')\n# [<b>The Dormouse's story</b>]\n
    "}, {"location": "beautifulsoup/#a-regular-expression", "title": "A regular expression", "text": "

    If you pass in a regular expression object, Beautiful Soup will filter against that regular expression using its search() method. This code finds all the tags whose names start with the letter b; in this case, the <body> tag and the <b> tag:

    import re\nfor tag in soup.find_all(re.compile(\"^b\")):\n    print(tag.name)\n# body\n# b\n
    "}, {"location": "beautifulsoup/#a-list", "title": "A list", "text": "

    If you pass in a list, Beautiful Soup will allow a string match against any item in that list. This code finds all the <a> tags and all the <b> tags:

    soup.find_all([\"a\", \"b\"])\n# [<b>The Dormouse's story</b>,\n#  <a class=\"sister\" href=\"http://example.com/elsie\" id=\"link1\">Elsie</a>,\n#  <a class=\"sister\" href=\"http://example.com/lacie\" id=\"link2\">Lacie</a>,\n#  <a class=\"sister\" href=\"http://example.com/tillie\" id=\"link3\">Tillie</a>]\n
    "}, {"location": "beautifulsoup/#a-function", "title": "A function", "text": "

    If none of the other matches work for you, define a function that takes an element as its only argument. The function should return True if the argument matches, and False otherwise.

    Here\u2019s a function that returns True if a tag defines the class attribute but doesn\u2019t define the id attribute:

    def has_class_but_no_id(tag):\n    return tag.has_attr('class') and not tag.has_attr('id')\n

    Pass this function into find_all() and you\u2019ll pick up all the <p> tags:

    soup.find_all(has_class_but_no_id)\n# [<p class=\"title\"><b>The Dormouse's story</b></p>,\n#  <p class=\"story\">Once upon a time there were...</p>,\n#  <p class=\"story\">...</p>]\n
    "}, {"location": "beautifulsoup/#find_all", "title": "find_all()", "text": "

    The find_all() method looks through a tag\u2019s descendants and retrieves all descendants that match your filters.

    soup.find_all(\"title\")\n# [<title>The Dormouse's story</title>]\n\nsoup.find_all(\"p\", \"title\")\n# [<p class=\"title\"><b>The Dormouse's story</b></p>]\n\nsoup.find_all(\"a\")\n# [<a class=\"sister\" href=\"http://example.com/elsie\" id=\"link1\">Elsie</a>,\n#  <a class=\"sister\" href=\"http://example.com/lacie\" id=\"link2\">Lacie</a>,\n#  <a class=\"sister\" href=\"http://example.com/tillie\" id=\"link3\">Tillie</a>]\n\nsoup.find_all(id=\"link2\")\n# [<a class=\"sister\" href=\"http://example.com/lacie\" id=\"link2\">Lacie</a>]\n\nimport re\nsoup.find(string=re.compile(\"sisters\"))\n# u'Once upon a time there were three little sisters; and their names were\\n'\n
    "}, {"location": "beautifulsoup/#the-name-argument", "title": "The name argument", "text": "

    Pass in a value for name and you\u2019ll tell Beautiful Soup to only consider tags with certain names. Text strings will be ignored, as will tags whose names that don\u2019t match.

    This is the simplest usage:

    soup.find_all(\"title\")\n# [<title>The Dormouse's story</title>]\n
    "}, {"location": "beautifulsoup/#the-keyword-arguments", "title": "The keyword arguments", "text": "

    Any argument that\u2019s not recognized will be turned into a filter on one of a tag\u2019s attributes. If you pass in a value for an argument called id, Beautiful Soup will filter against each tag\u2019s id attribute:

    soup.find_all(id='link2')\n# [<a class=\"sister\" href=\"http://example.com/lacie\" id=\"link2\">Lacie</a>]\n

    You can filter an attribute based on a string, a regular expression, a list, a function, or the value True.

    You can filter multiple attributes at once by passing in more than one keyword argument:

    soup.find_all(href=re.compile(\"elsie\"), id='link1')\n# [<a class=\"sister\" href=\"http://example.com/elsie\" id=\"link1\">three</a>]\n
    "}, {"location": "beautifulsoup/#searching-by-css-class", "title": "Searching by CSS class", "text": "

    It\u2019s very useful to search for a tag that has a certain CSS class, but the name of the CSS attribute, class, is a reserved word in Python. Using class as a keyword argument will give you a syntax error. As of Beautiful Soup 4.1.2, you can search by CSS class using the keyword argument class_:

    soup.find_all(\"a\", class_=\"sister\")\n# [<a class=\"sister\" href=\"http://example.com/elsie\" id=\"link1\">Elsie</a>,\n#  <a class=\"sister\" href=\"http://example.com/lacie\" id=\"link2\">Lacie</a>,\n#  <a class=\"sister\" href=\"http://example.com/tillie\" id=\"link3\">Tillie</a>]\n
    "}, {"location": "beautifulsoup/#the-string-argument", "title": "The string argument", "text": "

    With string you can search for strings instead of tags.

    soup.find_all(string=\"Elsie\")\n# [u'Elsie']\n\nsoup.find_all(string=[\"Tillie\", \"Elsie\", \"Lacie\"])\n# [u'Elsie', u'Lacie', u'Tillie']\n\nsoup.find_all(string=re.compile(\"Dormouse\"))\n[u\"The Dormouse's story\", u\"The Dormouse's story\"]\n\ndef is_the_only_string_within_a_tag(s):\n    \"\"\"Return True if this string is the only child of its parent tag.\"\"\"\n    return (s == s.parent.string)\n\nsoup.find_all(string=is_the_only_string_within_a_tag)\n# [u\"The Dormouse's story\", u\"The Dormouse's story\", u'Elsie', u'Lacie', u'Tillie', u'...']\n

    Although string is for finding strings, you can combine it with arguments that find tags: Beautiful Soup will find all tags whose .string matches your value for string.

    soup.find_all(\"a\", string=\"Elsie\")\n# [<a href=\"http://example.com/elsie\" class=\"sister\" id=\"link1\">Elsie</a>]\n
    "}, {"location": "beautifulsoup/#searching-by-attribute-and-value", "title": "Searching by attribute and value", "text": "
    soup = BeautifulSoup(html)\nresults = soup.findAll(\"td\", {\"valign\" : \"top\"})\n
    "}, {"location": "beautifulsoup/#the-limit-argument", "title": "The limit argument", "text": "

    find_all() returns all the tags and strings that match your filters. This can take a while if the document is large. If you don\u2019t need all the results, you can pass in a number for limit.

    "}, {"location": "beautifulsoup/#the-recursive-argument", "title": "The recursive argument", "text": "

    If you call mytag.find_all(), Beautiful Soup will examine all the descendants of mytag. If you only want Beautiful Soup to consider direct children, you can pass in recursive=False.

    "}, {"location": "beautifulsoup/#calling-a-tag-is-like-calling-find_all", "title": "Calling a tag is like calling find_all()", "text": "

    Because find_all() is the most popular method in the Beautiful Soup search API, you can use a shortcut for it. If you treat the BeautifulSoup object or a Tag object as though it were a function, then it\u2019s the same as calling find_all() on that object. These two lines of code are equivalent:

    soup.find_all(\"a\")\nsoup(\"a\")\n
    "}, {"location": "beautifulsoup/#find", "title": "find()", "text": "

    find() is like find_all() but returning just one result.

    "}, {"location": "beautifulsoup/#find_parent-and-find_parents", "title": "find_parent() and find_parents()", "text": "

    These methods work their way up the tree, looking at a tag\u2019s (or a string\u2019s) parents.

    "}, {"location": "beautifulsoup/#find_next_siblings-and-find_next_sibling", "title": "find_next_siblings() and find_next_sibling()", "text": "

    These methods use .next_siblings to iterate over the rest of an element\u2019s siblings in the tree. The find_next_siblings() method returns all the siblings that match, and find_next_sibling() only returns the first one:

    first_link = soup.a\nfirst_link\n# <a class=\"sister\" href=\"http://example.com/elsie\" id=\"link1\">Elsie</a>\n\nfirst_link.find_next_siblings(\"a\")\n# [<a class=\"sister\" href=\"http://example.com/lacie\" id=\"link2\">Lacie</a>,\n#  <a class=\"sister\" href=\"http://example.com/tillie\" id=\"link3\">Tillie</a>]\n

    To go in the other direction you can use find_previous_siblings() and find_previous_sibling()

    "}, {"location": "beautifulsoup/#modifying-the-tree", "title": "Modifying the tree", "text": ""}, {"location": "beautifulsoup/#replace_with", "title": "replace_with", "text": "

    PageElement.replace_with() removes a tag or string from the tree, and replaces it with the tag or string of your choice:

    markup = '<a href=\"http://example.com/\">I linked to <i>example.com</i></a>'\nsoup = BeautifulSoup(markup)\na_tag = soup.a\n\nnew_tag = soup.new_tag(\"b\")\nnew_tag.string = \"example.net\"\na_tag.i.replace_with(new_tag)\n\na_tag\n# <a href=\"http://example.com/\">I linked to <b>example.net</b></a>\n

    Sometimes it doesn't work. If it doesn't use:

    a_tag.clear()\na_tag.append(new_tag)\n
    "}, {"location": "beautifulsoup/#tips", "title": "Tips", "text": ""}, {"location": "beautifulsoup/#show-content-beautified-prettified", "title": "Show content beautified / prettified", "text": "

    Use print(soup.prettify()).

    "}, {"location": "beautifulsoup/#cleaning-escaped-html-code", "title": "Cleaning escaped HTML code", "text": "
    soup = BeautifulSoup(s.replace(r\"\\\"\", '\"').replace(r\"\\/\", \"/\"), \"html.parser\")\n
    "}, {"location": "beautifulsoup/#references", "title": "References", "text": ""}, {"location": "beets/", "title": "Beets", "text": "

    Beets is a music management library used to get your music collection right once and for all. It catalogs your collection, automatically improving its metadata as it goes using the MusicBrainz database. Then it provides a set of tools for manipulating and accessing your music.

    Through plugins it supports:

    Still, if beets doesn't do what you want yet, writing your own plugin is easy if you know a little Python. Or you can use it as a library.

    "}, {"location": "beets/#installation", "title": "Installation", "text": "
    pipx install beets\n

    You\u2019ll want to set a few basic options before you start using beets. The configuration is stored in a text file. You can show its location by running beet config -p, though it may not exist yet. Run beet config -e to edit the configuration in your favorite text editor. The file will start out empty, but here\u2019s good place to start:

    # Path to a directory where you\u2019d like to keep your music.\ndirectory: ~/music\n\n# Database file that keeps an index of your music.\nlibrary: ~/data/musiclibrary.db\n

    The default configuration assumes you want to start a new organized music folder (that directory above) and that you\u2019ll copy cleaned-up music into that empty folder using beets\u2019 import command. But you can configure beets to behave many other ways:

    "}, {"location": "beets/#usage", "title": "Usage", "text": ""}, {"location": "beets/#importing-your-library", "title": "Importing your library", "text": "

    The next step is to import your music files into the beets library database. Because this can involve modifying files and moving them around, data loss is always a possibility, so now would be a good time to make sure you have a recent backup of all your music. We\u2019ll wait.

    There are two good ways to bring your existing library into beets. You can either: (a) quickly bring all your files with all their current metadata into beets\u2019 database, or (b) use beets\u2019 highly-refined autotagger to find canonical metadata for every album you import. Option (a) is really fast, but option (b) makes sure all your songs\u2019 tags are exactly right from the get-go. The point about speed bears repeating: using the autotagger on a large library can take a very long time, and it\u2019s an interactive process. So set aside a good chunk of time if you\u2019re going to go that route.

    If you\u2019ve got time and want to tag all your music right once and for all, do this:

    $ beet import /path/to/my/music\n

    (Note that by default, this command will copy music into the directory you specified above. If you want to use your current directory structure, set the import.copy config option.) To take the fast, un-autotagged path, just say:

    $ beet import -A /my/huge/mp3/library\n

    Note that you just need to add -A for \u201cdon\u2019t autotag\u201d.

    "}, {"location": "beets/#references", "title": "References", "text": ""}, {"location": "book_binding/", "title": "Book binding", "text": "

    Book binding is the process of physically assembling a book of codex format from an ordered stack of paper sheets that are folded together into sections called signatures or sometimes left as a stack of individual sheets. Several signatures are then bound together along one edge with a thick needle and sturdy thread.

    "}, {"location": "book_binding/#references", "title": "References", "text": ""}, {"location": "book_binding/#videos", "title": "Videos", "text": ""}, {"location": "book_management/", "title": "Book Management", "text": "

    Book management is the set of systems and processes to get and categorize books so it's easy to browse and discover new content. It involves the next actions:

    I haven't yet found a single piece of software that fulfills all these needs, so we need to split it into subsystems.

    "}, {"location": "book_management/#downloader-and-indexer", "title": "Downloader and indexer", "text": "

    System that monitors the availability of books in a list of indexers, when they are available, they download it to a directory of the server. The best one that I've found is Readarr, it makes it easy to search for authors and books, supports a huge variety of indexers (such as Archive.org), and download clients (such as torrent clients).

    It can be used as a limited gallery browser, you can easily see the books of an author or series, but it doesn't yet support the automatic fetch of genres or tags.

    I haven't found an easy way of marking elements as read, prioritize the list of books to read, or add a user rating. Until these features are added (if they ever are), we need to use it in parallel with a better gallery browser.

    "}, {"location": "book_management/#gallery-browser", "title": "Gallery browser", "text": "

    System that shows the books in the library in a nice format, allowing the user to filter out the contents, prioritize them, mark them as read, rate them and optionally sync books with the ereader.

    Calibre-web is a beautiful solution, without trying it, it looks like it supports all of the required features, but it doesn't work well with Readarr. Readarr has support to interact with Calibre content server by defining a root folder to be managed by Calibre, but the books you want to have Readarr recognize on initial library import must already be in Calibre. Books within the folder and not in Calibre will be ignored. So you'll need to do the first import in Calibre, instead of Readarr (which is quite pleasant). Note also that you cannot add Calibre integration to a root folder after it's created.

    Calibre-web interacts directly with the Sqlite database of Calibre, so it doesn't expose the Calibre Content Server, therefore is not compatible with Readarr.

    To make it work, you'd need to have both the calibre server and the calibre-web running at the same time, which has led to database locks (1, and 2) that the calibre-web developer has tried to avoid by controlling the database writes, and said that:

    If you start Calibre first and afterwards Calibre-Web, Calibre indeed locks the database and doesn't allow Calibre-Web to access the database (metadata.db) file. Starting Calibre-Web and afterwards Calibre should work.

    The problem comes when Readarr writes in the database through calibre to add books, and calibre-web tries to write too to add user ratings or other metadata.

    Another option would be to only run calibre-web and automatically import the books once they are downloaded by Readarr. calibre-web is not going to support a watch directory feature, the author recommends to use a cron script to do it. I haven't tried this path yet.

    Another option would be to assume that calibre-web is not going to do any insert in the database, so it would become a read-only web interface, therefore we wouldn't be able to edit the books or rate them, one of the features we'd like to have in the gallery browser. To make sure that we don't get locks, instead of using the same file, a cron job could do an rsync between the database managed by calibre and the one used by calibre-web.

    Calibre implements genres with tags, behind the scenes it uses the fetch-ebook-metadata command line tool, that returns all the metadata in human readable form

    $: fetch-ebook-metadata -i 9780061796883 -c cover.jpg\n\nTitle               : The Dispossessed\nAuthor(s)           : Ursula K. le Guin\nPublisher           : Harper Collins\nTags                : Fiction, Science Fiction, Space Exploration, Literary, Visionary & Metaphysical\nLanguages           : eng\nPublished           : 2009-10-13T20:34:30.878865+00:00\nIdentifiers         : google:tlhFtmTixvwC, isbn:9780061796883\nComments            : \u201cOne of the greats\u2026.Not just a science fiction writer; a literary icon.\u201d \u2013 Stephen KingFrom the brilliant and award-winning author Ursula K. Le Guin comes a classic tale of two planets torn\napart by conflict and mistrust \u2014 and the man who risks everything to reunite them.A bleak moon settled by utopian anarchists, Anarres has long been isolated from other worlds, including its mother planet, Urras\u2014a\n civilization of warring nations, great poverty, and immense wealth. Now Shevek, a brilliant physicist, is determined to reunite the two planets, which have been divided by centuries of distrust. He will seek ans\nwers, question the unquestionable, and attempt to tear down the walls of hatred that have kept them apart.To visit Urras\u2014to learn, to teach, to share\u2014will require great sacrifice and risks, which Shevek willingly\n accepts. But the ambitious scientist's gift is soon seen as a threat, and in the profound conflict that ensues, he must reexamine his beliefs even as he ignites the fires of change.\nCover               : cover.jpg\n

    Or in xml if you use the -o flag.

    I've checked if these tags could be automatically applied to Readarr, but their tags are meant only to be attached to Authors to apply metadata profiles. I've opened an issue to see if they plan to implement tags for books.

    It's a pity we are not going to use calibre-web as it also had support to sync the reading stats from Kobo.

    In the past I used gcstar and then polar bookshelf, but decided not to use them anymore for different reasons.

    In conclusion, the tools reviewed don't work as I need them to, some ugly patches could be applied and maybe it would work, but it clearly shows that they are not ready yet unless you want to invest time in it, and even if you did, it will be unstable. Until a better system shows up, I'm going to use Readarr to browse the books that I want to read, and add them to an ordered markdown file with sections as genres, not ideal, but robust as hell xD.

    "}, {"location": "book_management/#review-system", "title": "Review system", "text": "

    System to write reviews and rate books, if the gallery browser doesn't include it, we'll use an independent component.

    Until I find something better, I'm saving the title, author, genre, score, and review in a json file, so it's easy to import in the chosen component.

    "}, {"location": "book_management/#content-discovery", "title": "Content discovery", "text": "

    Recommendation system to analyze the user taste and suggest books that might like. Right now I'm monitoring the authors with Readarr to get notifications when they release a new book. I also manually go through goodreads and similar websites looking for similar books to the ones I liked.

    "}, {"location": "book_management/#deprecated-components", "title": "Deprecated components", "text": ""}, {"location": "book_management/#polar-bookself", "title": "Polar bookself", "text": "

    It was a very promising piece of software that went wrong :(. It had a nice interface built for incremental reading and studying with anki, and a nice tag system. It was a desktop application you installed in your computer, but since Polar 2.0 they moved into a cloud hosted service, with no possibility of self-hosting it, so you give them your books and all your data, a nasty turn of events.

    "}, {"location": "book_management/#gcstar", "title": "GCStar", "text": "

    The first free open source application for managing collections I used, it has an old looking desktop interface and is no longer maintained.

    "}, {"location": "bookwyrm/", "title": "Bookwyrm", "text": "

    Bookwyrm is a social network for tracking your reading, talking about books, writing reviews, and discovering what to read next. Federation allows BookWyrm users to join small, trusted communities that can connect with one another, and with other ActivityPub services like Mastodon and Pleroma.

    "}, {"location": "bookwyrm/#references", "title": "References", "text": ""}, {"location": "boto3/", "title": "Boto3", "text": "

    Boto3 is the AWS SDK for Python to create, configure, and manage AWS services, such as Amazon Elastic Compute Cloud (Amazon EC2) and Amazon Simple Storage Service (Amazon S3). The SDK provides an object-oriented API as well as low-level access to AWS services.

    "}, {"location": "boto3/#installation", "title": "Installation", "text": "
    pip install boto3\n
    "}, {"location": "boto3/#usage", "title": "Usage", "text": ""}, {"location": "boto3/#s3", "title": "S3", "text": ""}, {"location": "boto3/#list-the-files-of-a-bucket", "title": "List the files of a bucket", "text": "
    def list_s3_by_prefix(\n    bucket: str, key_prefix: str, max_results: int = 100\n) -> List[str]:\n    next_token = \"\"\n    all_keys = []\n    while True:\n        if next_token:\n            res = s3.list_objects_v2(\n                Bucket=bucket, ContinuationToken=next_token, Prefix=key_prefix\n            )\n        else:\n            res = s3.list_objects_v2(Bucket=bucket, Prefix=key_prefix)\n\n        if \"Contents\" not in res:\n            break\n\n        if res[\"IsTruncated\"]:\n            next_token = res[\"NextContinuationToken\"]\n        else:\n            next_token = \"\"\n\n        keys = [item[\"Key\"] for item in res[\"Contents\"]]\n\n        all_keys.extend(keys)\n\n        if not next_token:\n            break\n    return all_keys[-1 * max_results :]\n

    The boto3 doesn't have any way to sort the outputs of the bucket, you need to do them once you've loaded all the objects :S.

    "}, {"location": "boto3/#ec2", "title": "EC2", "text": ""}, {"location": "boto3/#run-ec2-instance", "title": "Run EC2 instance", "text": "

    Use the run_instances method of the ec2 client. Check their docs for the different configuration options. The required ones are MinCount and MaxCount.

    import boto3\n\nec2 = boto3.client('ec2')\ninstance = ec2.run_instances(MinCount=1, MaxCount=1)\n
    "}, {"location": "boto3/#get-instance-types", "title": "Get instance types", "text": "
    from pydantic import BaseModel\nimport boto3\n\nclass InstanceType(BaseModel):\n    \"\"\"Define model of the instance type.\n\n    Args:\n        id_: instance type name\n        cpu_vcores: Number of virtual cpus (cores * threads)\n        cpu_speed: Sustained clock speed in Ghz\n        ram: RAM memory in MiB\n        network_performance:\n        price: Hourly cost\n    \"\"\"\n\n    id_: str\n    cpu_vcores: int\n    cpu_speed: Optional[int] = None\n    ram: int\n    network_performance: str\n    price: Optional[float] = None\n\n    @property\n    def cpu(self) -> int:\n        \"\"\"Calculate the total Ghz available.\"\"\"\n        if self.cpu_speed is None:\n            return self.cpu_vcores\n        return self.cpu_vcores * self.cpu_speed\n\n\n\ndef get_instance_types() -> InstanceTypes:\n    \"\"\"Get the available instance types.\"\"\"\n    log.info(\"Retrieving instance types\")\n    instance_types: InstanceTypes = {}\n    for type_ in _ec2_instance_types(cpu_arch=\"x86_64\"):\n        instance = InstanceType(\n            id_=instance_type,\n            cpu_vcores=type_[\"VCpuInfo\"][\"DefaultVCpus\"],\n            ram=type_[\"MemoryInfo\"][\"SizeInMiB\"],\n            network_performance=type_[\"NetworkInfo\"][\"NetworkPerformance\"],\n            price=_ec2_price(instance_type),\n        )\n\n        with suppress(KeyError):\n            instance.cpu_speed = type_[\"ProcessorInfo\"][\"SustainedClockSpeedInGhz\"]\n\n        instance_types[type_[\"InstanceType\"]] = instance\n\n    return instance_types\n
    "}, {"location": "boto3/#get-instance-prices", "title": "Get instance prices", "text": "
    import json\nimport boto3\nfrom pkg_resources import resource_filename\n\ndef _ec2_price(\n    instance_type: str,\n    region_code: str = \"us-east-1\",\n    operating_system: str = \"Linux\",\n    preinstalled_software: str = \"NA\",\n    tenancy: str = \"Shared\",\n    is_byol: bool = False,\n) -> Optional[float]:\n    \"\"\"Get the price of an EC2 instance type.\"\"\"\n    log.debug(f\"Retrieving price of {instance_type}\")\n    region_name = _get_region_name(region_code)\n\n    if is_byol:\n        license_model = \"Bring your own license\"\n    else:\n        license_model = \"No License required\"\n\n    if tenancy == \"Host\":\n        capacity_status = \"AllocatedHost\"\n    else:\n        capacity_status = \"Used\"\n\n    filters = [\n        {\"Type\": \"TERM_MATCH\", \"Field\": \"termType\", \"Value\": \"OnDemand\"},\n        {\"Type\": \"TERM_MATCH\", \"Field\": \"capacitystatus\", \"Value\": capacity_status},\n        {\"Type\": \"TERM_MATCH\", \"Field\": \"location\", \"Value\": region_name},\n        {\"Type\": \"TERM_MATCH\", \"Field\": \"instanceType\", \"Value\": instance_type},\n        {\"Type\": \"TERM_MATCH\", \"Field\": \"tenancy\", \"Value\": tenancy},\n        {\"Type\": \"TERM_MATCH\", \"Field\": \"operatingSystem\", \"Value\": operating_system},\n        {\n            \"Type\": \"TERM_MATCH\",\n            \"Field\": \"preInstalledSw\",\n            \"Value\": preinstalled_software,\n        },\n        {\"Type\": \"TERM_MATCH\", \"Field\": \"licenseModel\", \"Value\": license_model},\n    ]\n\n    pricing_client = boto3.client(\"pricing\", region_name=\"us-east-1\")\n    response = pricing_client.get_products(ServiceCode=\"AmazonEC2\", Filters=filters)\n\n    for price in response[\"PriceList\"]:\n        price = json.loads(price)\n\n        for on_demand in price[\"terms\"][\"OnDemand\"].values():\n            for price_dimensions in on_demand[\"priceDimensions\"].values():\n                price_value = price_dimensions[\"pricePerUnit\"][\"USD\"]\n\n        return float(price_value)\n    return None\n\n\ndef _get_region_name(region_code: str) -> str:\n    \"\"\"Extract the region name from it's code.\"\"\"\n    endpoint_file = resource_filename(\"botocore\", \"data/endpoints.json\")\n\n    with open(endpoint_file, \"r\", encoding=\"UTF8\") as f:\n        endpoint_data = json.load(f)\n\n    region_name = endpoint_data[\"partitions\"][0][\"regions\"][region_code][\"description\"]\n    return region_name.replace(\"Europe\", \"EU\")\n
    "}, {"location": "boto3/#type-hints", "title": "Type hints", "text": "

    AWS library doesn't have working type hints -.-, so you either use Any or dive into the myriad of packages that implement them. I've so far tried boto3_type_annotations, boto3-stubs, and mypy_boto3_builder without success. Any it is for now...

    "}, {"location": "boto3/#testing", "title": "Testing", "text": "

    Programs that interact with AWS through boto3 create, change or get information on real AWS resources.

    When developing these programs, you don't want the testing framework to actually do those changes, as it might break things and cost you money. You need to find a way to intercept the calls to AWS and substitute them with the data their API would return. I've found three ways to achieve this:

    TL;DR

    Try to use moto, using the stubber as fallback option.

    Using unittest.mock forces you to know what the API is going to return and hardcode it in your tests. If the response changes, you need to update your tests, which is not good.

    moto is a library that allows you to easily mock out tests based on AWS infrastructure. It works well because it mocks out all calls to AWS automatically without requiring any dependency injection. The downside is that it goes behind boto3 so some of the methods you need to test won't be still implemented, that leads us to the third option.

    Botocore's Stubber is a class that allows you to stub out requests so you don't have to hit an endpoint to write tests. Responses are returned first in, first out. If operations are called out of order, or are called with no remaining queued responses, an error will be raised. It's like the first option but cleaner. If you go down this path, check adamj's post on testing S3.

    "}, {"location": "boto3/#moto", "title": "moto", "text": "

    moto's library lets you fictitiously create and change AWS resources as you normally do with the boto3 library. They mimic what the real methods do on fake objects.

    The Docs are awful though.

    "}, {"location": "boto3/#install", "title": "Install", "text": "
    pip install moto\n
    "}, {"location": "boto3/#simple-usage", "title": "Simple usage", "text": "

    To understand better how it works, I'm going to show you an understandable example, it's not the best way to use it though, go to the usage section for production ready usage.

    Imagine you have a function that you use to launch new ec2 instances:

    import boto3\n\n\ndef add_servers(ami_id, count):\n    client = boto3.client('ec2', region_name='us-west-1')\n    client.run_instances(ImageId=ami_id, MinCount=count, MaxCount=count)\n

    To test it we'd use:

    from . import add_servers\nfrom moto import mock_ec2\n\n@mock_ec2\ndef test_add_servers():\n    add_servers('ami-1234abcd', 2)\n\n    client = boto3.client('ec2', region_name='us-west-1')\n    instances = client.describe_instances()['Reservations'][0]['Instances']\n    assert len(instances) == 2\n    instance1 = instances[0]\n    assert instance1['ImageId'] == 'ami-1234abcd'\n

    The decorator @mock_ec2 tells moto to capture all boto3 calls to AWS. When we run the add_servers function to test, it will create the fake objects on the memory (without contacting AWS servers), and the client.describe_instances boto3 method returns the data of that fake data. Isn't it awesome?

    "}, {"location": "boto3/#usage_1", "title": "Usage", "text": "

    You can use it with decorators, context managers, directly or with pytest fixtures.

    Being a pytest fan, the last option looks the cleaner to me.

    To make sure that you don't change the real infrastructure, ensure that your tests have dummy environmental variables.

    File: tests/conftest.py

    @pytest.fixture()\ndef _aws_credentials() -> None:\n    \"\"\"Mock the AWS Credentials for moto.\"\"\"\n    os.environ[\"AWS_ACCESS_KEY_ID\"] = \"testing\"\n    os.environ[\"AWS_SECRET_ACCESS_KEY\"] = \"testing\"\n    os.environ[\"AWS_SECURITY_TOKEN\"] = \"testing\"\n    os.environ[\"AWS_SESSION_TOKEN\"] = \"testing\"\n\n\n@pytest.fixture()\ndef ec2(_aws_credentials: None) -> Any:\n    \"\"\"Configure the boto3 EC2 client.\"\"\"\n    with mock_ec2():\n        yield boto3.client(\"ec2\", region_name=\"us-east-1\")\n

    The ec2 fixture can then be used in the tests to setup the environment or assert results.

    "}, {"location": "boto3/#testing-ec2", "title": "Testing EC2", "text": "

    If you want to add security groups to the tests, you need to create the resource first.

    def test_ec2_with_security_groups(ec2: Any) -> None:\n    security_group_id = ec2.create_security_group(\n        GroupName=\"TestSecurityGroup\", Description=\"SG description\"\n    )[\"GroupId\"]\n    instance = ec2.run_instances(\n        ImageId=\"ami-xxxx\",\n        MinCount=1,\n        MaxCount=1,\n        SecurityGroupIds=[security_group_id],\n    )[\"Instances\"][0]\n\n    # Test your code here\n

    To add tags, use:

    def test_ec2_with_security_groups(ec2: Any) -> None:\n    instance = ec2.run_instances(\n        ImageId=\"ami-xxxx\",\n        MinCount=1,\n        MaxCount=1,\n        TagSpecifications=[\n            {\n                \"ResourceType\": \"instance\",\n                \"Tags\": [\n                    {\n                        \"Key\": \"Name\",\n                        \"Value\": \"instance name\",\n                    },\n                ],\n            }\n        ],\n    )[\"Instances\"][0]\n\n    # Test your code here\n
    "}, {"location": "boto3/#testing-rds", "title": "Testing RDS", "text": "

    Use the rds fixture:

    from moto import mock_rds2\n\n@pytest.fixture()\ndef rds(_aws_credentials: None) -> Any:\n    \"\"\"Configure the boto3 RDS client.\"\"\"\n    with mock_rds2():\n        yield boto3.client(\"rds\", region_name=\"us-east-1\")\n

    To create an instance use:

    instance = rds.create_db_instance(\n    DBInstanceIdentifier=\"db-xxxx\",\n    DBInstanceClass=\"db.m3.2xlarge\",\n    Engine=\"postgres\",\n)[\"DBInstance\"]\n

    It won't have VPC information, if you need it, create the subnet group first (you'll need the ec2 fixture too):

    subnets = [subnet['SubnetId'] for subnet in ec2.describe_subnets()[\"Subnets\"]]\nrds.create_db_subnet_group(DBSubnetGroupName=\"dbsg\", SubnetIds=subnets, DBSubnetGroupDescription=\"Text\")\ninstance = rds.create_db_instance(\n    DBInstanceIdentifier=\"db-xxxx\",\n    DBInstanceClass=\"db.m3.2xlarge\",\n    Engine=\"postgres\",\n    DBSubnetGroupName=\"dbsg\",\n)[\"DBInstance\"]\n
    "}, {"location": "boto3/#testing-s3", "title": "Testing S3", "text": "

    Use the s3_mock fixture:

    from moto import mock_s3\n\n@pytest.fixture()\ndef s3_mock(_aws_credentials: None) -> Any:\n    \"\"\"Configure the boto3 S3 client.\"\"\"\n    with mock_s3():\n        yield boto3.client(\"s3\")\n

    To create an instance use:

    s3_mock.create_bucket(Bucket=\"mybucket\")\ninstance = s3_mock.list_buckets()[\"Buckets\"][0]\n

    Check the official docs to check the create_bucket arguments.

    "}, {"location": "boto3/#testing-route53", "title": "Testing Route53", "text": "

    Use the route53 fixture:

    from moto import mock_route53\n\n@pytest.fixture(name='route53')\ndef route53_(_aws_credentials: None) -> Any:\n    \"\"\"Configure the boto3 Route53 client.\"\"\"\n    with mock_route53():\n        yield boto3.client(\"route53\")\n

    To create an instance use:

    hosted_zone = route53.create_hosted_zone(\n    Name=\"example.com\", CallerReference=\"Test\"\n)[\"HostedZone\"]\nhosted_zone_id = re.sub(\".hostedzone.\", \"\", hosted_zone[\"Id\"])\nroute53.change_resource_record_sets(\n    ChangeBatch={\n        \"Changes\": [\n            {\n                \"Action\": \"CREATE\",\n                \"ResourceRecordSet\": {\n                    \"Name\": \"example.com\",\n                    \"ResourceRecords\": [\n                        {\n                            \"Value\": \"192.0.2.44\",\n                        },\n                    ],\n                    \"TTL\": 60,\n                    \"Type\": \"A\",\n                },\n            },\n        ],\n        \"Comment\": \"Web server for example.com\",\n    },\n    HostedZoneId=hosted_zone_id,\n)\n
    You need to first create a hosted zone. The change_resource_record_sets order to create the instance doesn't return any data, so if you need to work on it, use the list_resource_record_sets method of the route53 client (you'll need to set the HostedZoneId argument). If you have more than 300 records, the endpoint gives you a paginated response, so if the IsTruncated attribute is True, you need to call the method again setting the StartRecordName and StartRecordType to the NextRecordName and NextRecordType response arguments. Not nice at all.

    Pagination is not yet supported by moto, so you won't be able to test that part of your code.

    Check the official docs to check the method arguments:

    "}, {"location": "boto3/#test-vpc", "title": "Test VPC", "text": "

    Use the ec2 fixture defined in the usage section.

    To create an instance use:

    instance = ec2.create_vpc(\n    CidrBlock=\"172.16.0.0/16\",\n)[\"Vpc\"]\n

    Check the official docs to check the method arguments:

    "}, {"location": "boto3/#testing-autoscaling-groups", "title": "Testing autoscaling groups", "text": "

    Use the autoscaling fixture:

    from moto import mock_autoscaling\n\n@pytest.fixture(name='autoscaling')\ndef autoscaling_(_aws_credentials: None) -> Any:\n    \"\"\"Configure the boto3 Autoscaling Group client.\"\"\"\n    with mock_autoscaling():\n        yield boto3.client(\"autoscaling\")\n

    They don't yet support LaunchTemplates, so you'll have to use LaunchConfigurations. To create an instance use:

    autoscaling.create_launch_configuration(LaunchConfigurationName='LaunchConfiguration', ImageId='ami-xxxx', InstanceType='t2.medium')\nautoscaling.create_auto_scaling_group(AutoScalingGroupName='ASG name', MinSize=1, MaxSize=3, LaunchConfigurationName='LaunchConfiguration', AvailabilityZones=['us-east-1a'])\ninstance = autoscaling.describe_auto_scaling_groups()[\"AutoScalingGroups\"][0]\n

    Check the official docs to check the method arguments:

    "}, {"location": "boto3/#test-security-groups", "title": "Test Security Groups", "text": "

    Use the ec2 fixture defined in the usage section.

    To create an instance use:

    instance_id = ec2.create_security_group(\n    GroupName=\"TestSecurityGroup\", Description=\"SG description\"\n)[\"GroupId\"]\ninstance = ec2.describe_security_groups(GroupIds=[instance_id])\n

    To add permissions to the security group you need to use the authorize_security_group_ingress and authorize_security_group_egress methods.

    ec2.authorize_security_group_ingress(\n    GroupId=instance_id,\n    IpPermissions=[\n        {\n            \"IpProtocol\": \"tcp\",\n            \"FromPort\": 80,\n            \"ToPort\": 80,\n            \"IpRanges\": [{\"CidrIp\": \"0.0.0.0/0\"}],\n        },\n    ],\n)\n

    By default, the created security group comes with an egress rule to allow all traffic. To remove rules use the revoke_security_group_egress and revoke_security_group_ingress methods.

    ec2.revoke_security_group_egress(\n    GroupId=instance_id,\n    IpPermissions=[\n        {\"IpProtocol\": \"-1\", \"IpRanges\": [{\"CidrIp\": \"0.0.0.0/0\"}]},\n    ],\n)\n

    Check the official docs to check the method arguments:

    "}, {"location": "boto3/#test-iam-users", "title": "Test IAM users", "text": "

    Use the iam fixture:

    from moto import mock_iam\n\n@pytest.fixture(name='iam')\ndef iam_(_aws_credentials: None) -> Any:\n    \"\"\"Configure the boto3 IAM client.\"\"\"\n    with mock_iam():\n        yield boto3.client(\"iam\")\n

    To create an instance use:

    instance = iam.create_user(UserName=\"User\")[\"User\"]\n

    Check the official docs to check the method arguments:

    "}, {"location": "boto3/#test-iam-groups", "title": "Test IAM Groups", "text": "

    Use the iam fixture defined in the test IAM users section:

    To create an instance use:

    user = iam.create_user(UserName=\"User\")[\"User\"]\ninstance = iam.create_group(GroupName=\"UserGroup\")[\"Group\"]\niam.add_user_to_group(GroupName=instance[\"GroupName\"], UserName=user[\"UserName\"])\n

    Check the official docs to check the method arguments:

    "}, {"location": "boto3/#issues", "title": "Issues", "text": ""}, {"location": "boto3/#references", "title": "References", "text": ""}, {"location": "calendar_management/", "title": "Calendar Management", "text": "

    Since the break of my taskwarrior instance I've used a physical calendar to manage the tasks that have a specific date.

    The next factors made me search for a temporal solution:

    To fulfill my needs the solution needs to:

    "}, {"location": "calendar_management/#khal", "title": "Khal", "text": "

    Looking at the available programs I found khal, which looks like it may be up to the task.

    Go through the installation steps and configure the instance to have a local calendar.

    If you want to sync your calendar events through CalDAV, you need to set vdirsyncer.

    "}, {"location": "calendar_versioning/", "title": "Calendar Versioning", "text": "

    Calendar Versioning is a versioning convention based on your project's release calendar, instead of arbitrary numbers.

    CalVer suggests version number to be in format of: YEAR.MONTH.sequence. For example, 20.1 indicates a release in 2020 January, while 20.5.2 indicates a release that occurred in 2020 May, while the 2 indicates this is the third release of the month.

    You can see it looks similar to semantic versioning and has the benefit that a later release qualifies as bigger than an earlier one within the semantic versioning world (which mandates that a version number must grow monotonically). This makes it easy to use in all places where semantic versioning can be used.

    The idea here is that if the only maintained version is the latest, then we might as well use the version number to indicate the release date to signify just how old of a version you\u2019re using. You also have the added benefit that you can make calendar-based promises. For example, Ubuntu offers five years of support, therefore given version 20.04 you can quickly determine that it will be supported up to April 2025.

    "}, {"location": "calendar_versioning/#when-to-use-calver", "title": "When to use CalVer", "text": "

    Check the Deciding what version system to use for your programs article section.

    "}, {"location": "calendar_versioning/#references", "title": "References", "text": ""}, {"location": "changelog/", "title": "Changelog", "text": "

    A changelog is a file which contains a curated, chronologically ordered list of notable changes for each version of a project.

    It's purpose is to make it easier for users and contributors to see precisely what notable changes have been made between each release (or version) of the project.

    "}, {"location": "changelog/#types-of-changes", "title": "Types of changes", "text": ""}, {"location": "changelog/#changelog-guidelines", "title": "Changelog Guidelines", "text": "

    Good changelogs follow the next principles:

    Some examples of bad changelogs are:

    "}, {"location": "changelog/#how-to-reduce-the-effort-required-to-maintain-a-changelog", "title": "How to reduce the effort required to maintain a changelog", "text": "

    There are two ways to ease the burden of maintaining a changelog:

    "}, {"location": "changelog/#build-it-automatically", "title": "Build it automatically", "text": "

    If you use Semantic Versioning you can use the commitizen tool to automatically generate the changelog each time you cut a new release by running cz bump --changelog --no-verify.

    The --no-verify part is required if you use pre-commit hooks.

    "}, {"location": "changelog/#use-the-unreleased-section", "title": "Use the Unreleased section", "text": "

    Keep an Unreleased section at the top to track upcoming changes.

    This serves two purposes:

    "}, {"location": "changelog/#references", "title": "References", "text": ""}, {"location": "chezmoi/", "title": "Chezmoi", "text": "

    Chezmoi stores the desired state of your dotfiles in the directory ~/.local/share/chezmoi. When you run chezmoi apply, chezmoi calculates the desired contents for each of your dotfiles and then makes the minimum changes required to make your dotfiles match your desired state.

    What I like:

    What I don't like:

    "}, {"location": "chezmoi/#installation", "title": "Installation", "text": "

    I've added some useful aliases:

    alias ce='chezmoi edit'\nalias ca='chezmoi add'\nalias cdiff='chezmoi diff'\nalias cdata='chezmoi edit-config'\nalias capply='chezmoi apply'\nalias cexternal='nvim ~/.local/share/chezmoi/.chezmoiexternal.yaml'\n
    "}, {"location": "chezmoi/#basic-usage", "title": "Basic Usage", "text": ""}, {"location": "chezmoi/#first-install", "title": "First install", "text": "

    Assuming that you have already installed chezmoi, initialize chezmoi with:

    $ chezmoi init\n

    This will create a new git local repository in ~/.local/share/chezmoi where chezmoi will store its source state. By default, chezmoi only modifies files in the working copy.

    Manage your first file with chezmoi:

    $ chezmoi add ~/.bashrc\n

    This will copy ~/.bashrc to ~/.local/share/chezmoi/dot_bashrc.

    Edit the source state:

    $ chezmoi edit ~/.bashrc\n

    This will open ~/.local/share/chezmoi/dot_bashrc in your $EDITOR. Make some changes and save the file.

    See what changes chezmoi would make:

    $ chezmoi diff\n

    Apply the changes:

    $ chezmoi -v apply\n

    Sometimes the diff is too big and you need to work with it chuck by chunk. For each change you can either:

    Here <target> is any directory or file listed in the diff.

    All chezmoi commands accept the -v (verbose) flag to print out exactly what changes they will make to the file system, and the -n (dry run) flag to not make any actual changes. The combination -n -v is very useful if you want to see exactly what changes would be made.

    Next, open a shell in the source directory, to commit your changes:

    $ chezmoi cd\n$ git add .\n$ git commit -m \"Initial commit\"\n

    Create a new repository on your desired git server called dotfiles and then push your repo:

    $ git remote add origin https://your_git_server.com/$GIT_USERNAME/dotfiles.git\n$ git branch -M main\n$ git push -u origin main\n

    Hint: chezmoi can be configured to automatically add, commit, and push changes to your repo.

    Finally, exit the shell in the source directory to return to where you were:

    $ exit\n
    "}, {"location": "chezmoi/#install-a-binary-from-an-external-url", "title": "Install a binary from an external url", "text": "

    Sometimes you may want to install some binaries from external urls, for example velero a backup tool for kubernetes. And you may want to be able to define what version you want to have and be able to update it at will.

    To do that we can define the version in the configuration with chezmoi edit-config

    data:\n  velero_version: 1.9.5\n

    All the variables you define under the data field are globally available on all your templates.

    Then we can set the external configuration of chezmoi by editing the file ~/.config/chezmoi/.chezmoiexternal.yaml and add the next snippet:

    .local/bin/velero:\n  type: \"file\"\n  url: https://github.com/vmware-tanzu/velero/releases/download/v{{ .velero_version }}/velero-v{{ .velero_version }}-{{ .chezmoi.os }}-{{ .chezmoi.arch }}.tar.gz\n  executable: true\n  refreshPeriod: 168h\n  filter:\n    command: tar\n    args:\n      - --extract\n      - --file\n      - /dev/stdin\n      - --gzip\n      - --to-stdout\n      - velero-v{{ .velero_version }}-{{ .chezmoi.os }}-{{ .chezmoi.arch }}/velero\n

    This will download the binary of version 1.9.5 from the source, unpack it and extract the velero binary and save it to ~/.local/bin/velero.

    "}, {"location": "chezmoi/#references", "title": "References", "text": ""}, {"location": "chromium/", "title": "Chromium", "text": "
    apt-get install chromium\n
    "}, {"location": "code_learning/", "title": "Learning to code", "text": "

    Learning to code is a never ending, rewarding, frustrating, enlightening task. In this article you can see what is the generic roadmap (in my personal opinion) of a developer. As each of us is different, probably a generic roadmap won't suit your needs perfectly, if you are new to coding, I suggest you find a mentor so you can both tweak it to your case.

    "}, {"location": "code_learning/#learning-methods", "title": "Learning methods", "text": "

    Not all of us like to learn in the same way, first you need to choose how do you want to learn, for example through:

    Whichever you choose make sure you have regular feedback from other humans such as:

    "}, {"location": "code_learning/#roadmap", "title": "Roadmap", "text": "

    The roadmap is divided in these main phases:

    Don't try to rush, this is a lifetime roadmap, depending on how much time you put into learning the first steps may take from months to one or two years, the refinement phase from 2 to 8 years, and the enhancement phase never ends.

    "}, {"location": "code_learning/#beginner", "title": "Beginner", "text": "

    First steps are hard, you're entering a whole new world that mostly looks like magic to you. Probably you'll feel overwhelmed by the path ahead but don't fret, as every path, it's doable one step at a time.

    First steps are also exciting so try to channel all that energy into the learning process to overcome the obstacles you find in your way.

    In this section you'll learn how to start walking in the development world by:

    "}, {"location": "code_learning/#setup-your-development-environment", "title": "Setup your development environment", "text": ""}, {"location": "code_learning/#editor", "title": "Editor", "text": "

    TBD

    "}, {"location": "code_learning/#git", "title": "Git", "text": "

    Git is a software for tracking changes in any set of files, usually used for coordinating work among programmers collaboratively developing source code during software development. Its goals include speed, data integrity, and support for distributed, non-linear workflows (thousands of parallel branches running on different systems).

    Git is a tough nut to crack, no matter how experience you are you'll frequently get surprised. Sadly it's one of the main tools to develop your code, so you must master it as soon as possible.

    I've listed you some resources here on how to start. From that article I think it's also interesting that you read about:

    "}, {"location": "code_learning/#language-specific-environment", "title": "Language specific environment", "text": ""}, {"location": "code_learning/#learn-the-basics", "title": "Learn the basics", "text": "

    Now it's the time to study, choose your desired learning method and follow them until you get the basics of the type of developer you want to become, for example:

    In parallel it's crucial to learn Git as soon as you can, it's the main tool to collaborate with other developers and your safety net in the development workflow.

    "}, {"location": "code_learning/#searching-for-information", "title": "Searching for information", "text": ""}, {"location": "code_learning/#junior", "title": "Junior", "text": "

    TBD

    "}, {"location": "code_learning/#senior", "title": "Senior", "text": "

    TBD

    "}, {"location": "collaborating_tools/", "title": "Collaborating tools", "text": ""}, {"location": "collaborating_tools/#collaborating-document-creation", "title": "Collaborating document creation", "text": ""}, {"location": "collaborating_tools/#collaborating-through-terminals", "title": "Collaborating through terminals", "text": ""}, {"location": "cone/", "title": "Cone", "text": "

    Cone is a mobile ledger application compatible with beancount. I use it as part of my accounting automation workflow.

    "}, {"location": "cone/#installation", "title": "Installation", "text": ""}, {"location": "cone/#usage", "title": "Usage", "text": "

    To be compliant with my beancount ledger:

    If I need to edit or delete a transaction, I change it with the Markor editor.

    To send the ledger file to the computer, I use either Share via HTTP or Termux through ssh.

    "}, {"location": "cone/#references", "title": "References", "text": ""}, {"location": "configuration_management/", "title": "Configuration management", "text": "

    Configuring your devices is boring, disgusting and complex. Specially when your device dies and you need to reinstall. You usually don't have the time or energy to deal with it, you just want it to work.

    To have a system that allows you to recover from a disaster it's expensive in both time and knowledge, and many people have different solutions.

    This article shows the latest step of how I'm doing it.

    "}, {"location": "configuration_management/#linux-devices", "title": "Linux devices", "text": ""}, {"location": "configuration_management/#operating-system-installation", "title": "Operating System installation", "text": "

    I don't use PEX or anything, just burn the ISO in an USB and manually install it.

    "}, {"location": "configuration_management/#main-skeleton", "title": "Main skeleton", "text": "

    I use the same Ansible playbook to configure all my devices. Using a monolith playbook is usually not a good idea in enterprise environments, you'd better use a cruft template and a playbook for each server. At your home you'd probably have a limited number of devices and maintaining many playbooks and a cruft template is not worth your time. Ansible is flexible enough to let you manage the differences of the devices while letting you reuse common parts.

    The playbook configures the skeleton of the device namely:

    In that playbook I use roles in these ways:

    "}, {"location": "configuration_management/#home-configuration", "title": "Home configuration", "text": ""}, {"location": "contact/", "title": "Contact", "text": "

    I'm available through:

    PGP Key: 6ADA882386CDF9BD1884534C6C7D7C1612CDE02F 
    -----BEGIN PGP PUBLIC KEY BLOCK-----\n\nmQINBFhs5wUBEAC289UxruAPfjvJ723AKhUhRI0/fw+cG0IeSUJfOSvWW+HJ7Elo\nQoPkKYv6E1k4SzIt6AgbEWpL35PQP79aQ5BFog2SbfVvfnq1/gIasFlyeFX1BUTh\nzxTKrYKwbUdsTeMYw32v5p2Q+D8CZK6/0RCM/GSb5oMPVancOeoZs8IebKpJH2x7\nHCniyQbq7xiFU5sUyB6tmgCiXg8INib+oTZqGKW/sVaxmTdH+fF9a2nnH0TN8h2W\n5V5XQ9/VQZk/GHQVq/Y0Z73BibOJM5Bv+3r2EIJfozlpWdUblat45lSATBo/sktf\nYKlxwAztWPtcTavJ58F1ufGcUPjwGW4E92zRaozC+tpzd5QtHeYM7m6fGlXxckua\nUesZcZLl9pY4Bc8Mw40WvI1ibhA2mP2R5AO8hJ0vJyFfi35lqM/DJVV1900yp+em\nuY+u6bNJ1gLLb7QnhbV1VYLTSCoWzPQvWHgMHAKpAjO15rKAItXD17BM2eQgJMuX\nLcoWeOcz/MrMQiGKSkqpmapwgtDZ5t81D2qWv+wsaZgcO/erknugHFmR3kAP8YHp\nJsIpaYY7kj+yVJb92uzZKQAEaUpq3uRsBDtkoC2MPzKN4fgWa8f4jBpIzxuBTd+6\n75sVq5VB5eaq3w4J0Z4kbk1DVyNffv3LeZCv9oC2mb1aXyVD/gWHlPD+6wARAQAB\ntBZseXouLiA8bHl6QHJpc2V1cC5uZXQ+iQJUBBMBCAA+AhsDBQsJCAcCBhUICQoL\nAgQWAgMBAh4BAheAFiEEatqII4bN+b0YhFNMbH18FhLN4C8FAl/oSy0FCQlcl6gA\nCgkQbH18FhLN4C/7Xg/+IyKUbaRwTYX+BqJ0kbi8auM0m+8oUFtFBK7FIzR860m/\nn+PDmKxLKHEi1yaa6gyylLHub3OSNI8DgiFD3eQiT3eKjotOaGwFK2lziynysO5k\nJdq3wAlzteoHXblUvtXt4RD7unBlE2LIlmq2KR2jNvtHbawhRNsjvcuft9rvkMot\n2qgZfZBixXKWYU0u9e+nucfR0dwlnc3kXnS/VFF3pfQBNLy515v6pA9TXUNMZ9+K\nZQeRRfIqVpr7Tu/4f0M4wPbW80cR1tcRz8vOOTHXi1+KsnpkjkwAYdlNmL7uziGQ\nxuhBlCrEM94OauuBvMkACysAOfor8SgpNYovuAOxYrojLwr7EBE+MI60cNpWP5tv\nQTC6kcATf3QFAtnm3INEfkh9s/7HSEsImWOwMTLlnWLTcto3Q2LLY5O6OX8ak0sv\niYRMAjHdoCp3bgWJl+wrrVV0G7VQMJFC895Z21JrUBBroGSyuefHOrkN13qsrO1K\nwJovpvlWkf5PzyJnIAuDASh7SvJaocOOD3zOCW/RxGY0W51+T2fCpGbNA54JJ2gj\nkplJo00ekXWGxv/fxguNgrtkV3F9GbubQ00SyXZ4tbRfgpZLOdjxblPjvPptIXIt\ntZxmQ7YWa/MyUBXLiwvDfxDYiGxEamqzrYZO4xiyM+PvQ+ZDlzEcjpFztNxmjnW5\nAg0EWGznBQEQALNL9sNc4SytS3fOcS4gHvZpH3TLJ6o0K/Lxg4RfkLMebDJwWvSW\nmjQLv3GqRfOhGj2Osi2YukFIJb4vxPJFO7wQhCi5LLSVEb5d/z9ZOJUdGdI9JvGW\ndFDuLEXwDnJaP5Jmjm3DwbvHK+goI7Fn3TKc27iqOVAKVIjWNPaqFZxwIE9o/+1c\n3bTk3A8WOBmcv1IaxsUNkRDOFJlQYLM/bFIuDD+cW/CcYro8ouC9aekmvTDoRaU5\nxv++fXtesn6Cy+xBgvBGIIXGo5xzd6Y66Yf8uNpuJXo9Dc6rApH1QEQNwZX1cxvG\nUpQx+9JNF0eptDLvTgmxcCglllrylcw8ZsVEt6BTgrCd2JXMGxUcAnhXpRWRmXNL\nn97FOBb6OBd6k7DC6QCiVKr7sytq1Ywl8GTtWrTP7sK+/+KDLPJ/oY7+bwV94+N8\nGthr94njNqb5G6t9fqQ/+cJv7oF8DoBvylYGqm2hvYpOH53hMq1y3OTPoFKP6AIx\ntwIWHkdmMALm6a6bxAetGQxiaPZTOduJDehwiF9EUkiNhpESMl3I2+vH86jV2IiT\n4BuUqGBU5wrAN/FixIRlmaSUX7e0OkUkDexVlpw5poJbPEbvhOtuj/V9BOxQKWB4\nbjXMHEHR5YcJ1lhPjFFM3pqOz6ZaN8Hs70KOBE+/3/c1hS5debWPBMdlABEBAAGJ\nAjwEGAEIACYCGwwWIQRq2ogjhs35vRiEU0xsfXwWEs3gLwUCX+hLMwUJCVyXrgAK\nCRBsfXwWEs3gL2JMEACZyzLv3IubRVR6cP5HyKbdAA4+fgzONwxw1TloC3T0okjN\nwHuhZJuv22GEaicklqHceIIYYssh7jER+plaDBA8t8xsvJ/7T3xMAt5RMsU2INWc\nKFkUkOmj3Q0aLnyVGKlrynMGnetMA+OliMnOrWWqDIomrfcJjdc3/OpCigR/lgCL\njItH4nH/NwjyexcQuKfpDgGcxUQhhPyfkyQrOxRBRXgjhB1Q3ra6MqM2g36EPLn3\nhSoQFbmUIbOYye6Vn7fENC5fmRS/4RcGGaq+wlAK0Mc1U8Bzl0AVBU3q+bKgDihO\ndRWEz9GV2UuDN7MhuUMSX3GFWIS/Gd0fc6EqDDHP0IWdwd268S5jTqvaz9IddQJ3\nvGPR++Vjex8VepCHsPBC2i7RmlBgbEvjWIHCEcBtyxd8TY7/3VFkzrQqY4bD0pyK\nl67QIP/ybgPqYgD0zfVyYa0oaZlk9OIH48SE4AwOVE7lTsEOWJ+EBodUtW094TFv\nnTZ2Uusuxg+rS6SbtDqcxvVBoSPYqtRSNS9FidWamuXudb6Ia/hLfBvDQxnzfaVR\nEGGpxmgzqLwKaGx7Cf5dnrfz2NVD9Mxb78n/Lk3qnQvD6CpzcdB+u4S+aWMzvuiN\nziOWBy/hZuRWttlZW2dN290w0csREWldUw7jkAWUCxLiMePCOVUj9cNDlPbPeQ==\n=QNxk\n-----END PGP PUBLIC KEY BLOCK-----\n
    "}, {"location": "cooking/", "title": "Cooking", "text": "

    Cooking as defined in Wikipedia, is the art, science, and craft of using heat to prepare food for consumption. It sounds like an enlightening experience that brings you joy. Reality then slaps you in the face yet once again. It's very different to cook because you want to, than cooking because you need to. I love to eat, but have always hated to cook, mainly because I've always seen cooking with that second view.

    Being something disgusting that I had to do, pushed me to batch cook once a week as quickly as possible, and to buy prepared food from the local squat center's tavern. Now I aim to shift my point of view to enjoy the time invested preparing food. Two are the main reasons: I'm going to spend a great amount of my life in front of the stove, so I'd better enjoy it, and probably the end result would be better for my well being.

    One way that can help me with the switch, is to understand the science behind it and be precise with the process. Thus this section was born, I'll start with the very basics and build from there on.

    "}, {"location": "cooking_basics/", "title": "Cooking Basics", "text": "

    All great recipes are based on the same basic principles and processes, these are the cooking snippets that I've needed.

    "}, {"location": "cooking_basics/#boiling-an-egg", "title": "Boiling an egg", "text": "

    Cooking an egg well is a matter of time.

    Here are some tips to improve your chances to get the perfect egg:

    "}, {"location": "cooking_basics/#cooking-legumes", "title": "Cooking legumes", "text": "

    Legumes are wonderful, to squeeze their value remember to:

    When you're using boiled legumes in your recipes, be careful, after the hydration, they weight the double!

    "}, {"location": "cooking_basics/#boil-chickpeas-when-youve-forgotten-to-soak-them", "title": "Boil chickpeas when you've forgotten to soak them", "text": "

    Soaked chickpeas take one or two hours to cook in a normal pot, 20 to 30 in a fast pot, which saves a ton of energy.

    If you forgot to soak them, add a level teaspoon of baking soda to the pot and cook them as usual. When not using a fast pot, you'll need to periodically remove the foam that will be created. The problem with this method is that you don't discard the first water round, and they can be more indigestible. Another option is to cook them for an hour, change the water and then cook them again.

    "}, {"location": "cooking_software/", "title": "Cooking software", "text": "

    While using grocy to manage my recipes I've found what I want from a recipe manager:

    "}, {"location": "cooking_software/#software-review", "title": "Software review", "text": ""}, {"location": "cooking_software/#grocy", "title": "Grocy", "text": "

    Grocy is an awesome software to manage your inventory, it's also a good one to manage your recipes, with an awesome integration with the inventory management. In fact, I've been using it for some years.

    Nevertheless, you can't:

    So I think whatever I choose to manage recipes needs to be able to speak to a Grocy instance at least to get an idea of the stock available and to complete the shopping list.

    "}, {"location": "cooking_software/#cooklang", "title": "Cooklang", "text": "

    Cooklang looks real good, you write the recipes in plaintext but the syntax is not as smooth as I'd like:

    Then add @salt and @ground black pepper{} to taste.\n

    I'd like the parser to be able to detect the ingredient salt without the need of the @, and I don't either like how to specify the measurements:

    Place @bacon strips{1%kg} on a baking sheet and glaze with @syrup{1/2%tbsp}.\n

    On the other side there is more less popular:

    It could be used as part of the system, but it falls short in many of my desired features.

    "}, {"location": "cooking_software/#kookbook", "title": "KookBook", "text": "

    KookBook is KDE solution for plaintext recipe management. Their documentation is sparse and not popular at all. I don't feel like using it.

    "}, {"location": "cooking_software/#recipesage", "title": "RecipeSage", "text": "

    RecipeSage is free personal recipe keeper, meal planner, and shopping list manager for Web, IOS, and Android.

    Quickly capture and save recipes from any website simply by entering the website URL. Sync your recipes, meal plans, and shopping lists between all of your devices. Share your recipes, shopping lists, and meal plans with family and friends.

    It looks good, but I'd use grocy instead.

    "}, {"location": "cooking_software/#mealie", "title": "Mealie", "text": "

    Mealie is a self hosted recipe manager and meal planner with a RestAPI backend and a reactive frontend application built in Vue for a pleasant user experience for the whole family. Easily add recipes into your database by providing the url and mealie will automatically import the relevant data or add a family recipe with the UI editor.

    It does have an API, but it looks too complex. Maybe to be used as a backend to retrieve recipes from the internet, but no plaintext recipes.

    "}, {"location": "cooking_software/#chowdown", "title": "Chowdown", "text": "

    Chowdown is a simple, plaintext recipe database for hackers. It has nice features:

    An example would be:

    ---\n\nlayout: recipe\ntitle:  \"Broccoli Beer Cheese Soup\"\nimage: broccoli-beer-cheese-soup.jpg\ntags: sides, soups\n\ningredients:\n- 4 tablespoons butter\n- 1 cup diced onion\n- 1/2 cup shredded carrot\n- 1/2 cup diced celery\n- 1 tablespoon garlic\n- 1/4 cup flour\n- 1 quart chicken broth\n- 1 cup heavy cream\n- 10 ounces muenster cheese\n- 1 cup white white wine\n- 1 cup pale beer\n- 1 teaspoon Worcestershire sauce\n- 1/2 teaspoon hot sauce\n\ndirections:\n- Start with butter, onions, carrots, celery, garlic until cooked down\n- Add flour, stir well, cook for 4-5 mins\n- Add chicken broth, bring to a boil\n- Add wine and reduce to a simmer\n- Add cream, cheese, Worcestershire, and hot sauce\n- Serve with croutons\n\n---\n\nThis recipe is inspired by one of my favorites, Gourmand's Beer Cheese Soup, which uses Shiner Bock. Feel free to use whatever you want, then go to [Gourmand's](http://lovethysandwich.com) to have the real thing.\n

    Or using other recipes:

    ---\n\nlayout: recipe\ntitle:  \"Red Berry Tart\"\nimage: red-berry-tart.jpg\ntags: desserts\n\ndirections:\n- Bake the crust and let it cool\n- Make the custard, pour into crust\n- Make the red berry topping, spread over the top\n\ncomponents:\n- Graham Cracker Crust\n- Vanilla Custard Filling\n- Red Berry Dessert Topping\n\n---\n\nA favorite when I go to BBQs (parties, hackathons, your folks' place), this red berry tart is fairly easy to make and packs a huge wow factor.\n

    Where Graham Cracker Crust is another recipe. The outcome is nice too.

    Downsides are:

    It redirects to an interesting schema of a recipe.

    https://github.com/clarklab/chowdown https://raw.githubusercontent.com/clarklab/chowdown/gh-pages/_recipes/broccoli-cheese-soup.md https://www.paprikaapp.com/

    "}, {"location": "cooking_software/#recipes", "title": "Recipes", "text": "

    https://docs.tandoor.dev/features/authentication/

    "}, {"location": "cooking_software/#chef", "title": "Chef", "text": ""}, {"location": "copier/", "title": "copier", "text": "

    Copier is a library and CLI app for rendering project templates.

    "}, {"location": "copier/#installation", "title": "Installation", "text": "
    pipx install copier\n

    Until this issue is solved you also need to downgrade pydantic

    pipx inject copier 'pydantic<2'\n
    "}, {"location": "copier/#basic-concepts", "title": "Basic concepts", "text": "

    Copier is composed of these main concepts:

    Copier targets these main human audiences:

    Non-humans should be happy also by using Copier's CLI or API, as long as their expectations are the same as for those humans... and as long as they have feelings.

    Templates have these goals:

    Copier tries to have a smooth learning curve that lets you create simple templates that can evolve into complex ones as needed.

    "}, {"location": "copier/#usage", "title": "Usage", "text": ""}, {"location": "copier/#creating-a-template", "title": "Creating a template", "text": "

    A template is a directory: usually the root folder of a Git repository.

    The content of the files inside the project template is copied to the destination without changes, unless they end with .jinja. In that case, the templating engine will be used to render them.

    Jinja2 templating is used. Learn more about it by reading Jinja2 documentation.

    If a YAML file named copier.yml or copier.yaml is found in the root of the project, the user will be prompted to fill in or confirm the default values.

    Minimal example:

    \ud83d\udcc1 my_copier_template                            # your template project\n\u251c\u2500\u2500 \ud83d\udcc4 copier.yml                                # your template configuration\n\u251c\u2500\u2500 \ud83d\udcc1 .git/                                     # your template is a Git repository\n\u251c\u2500\u2500 \ud83d\udcc1 {{project_name}}                          # a folder with a templated name\n\u2502   \u2514\u2500\u2500 \ud83d\udcc4 {{module_name}}.py.jinja              # a file with a templated name\n\u2514\u2500\u2500 \ud83d\udcc4 {{_copier_conf.answers_file}}.jinja       # answers are recorded here\n

    Where:

    Generating a project from this template using copier copy my_copier_template generated_project answering super_project and world for the project_name and module_name questions respectively would create in the following directory and files:

    \ud83d\udcc1 generated_project\n\u251c\u2500\u2500 \ud83d\udcc1 super_project\n\u2502   \u2514\u2500\u2500 \ud83d\udcc4 world.py\n\u2514\u2500\u2500 \ud83d\udcc4 .copier-answers.yml\n

    Where:

    "}, {"location": "copier/#questions", "title": "questions", "text": "

    project_name: type: str help: What is your project name?

    module_name: type: str help: What is your Python module name? ```

    "}, {"location": "copier/#template-helpers", "title": "Template helpers", "text": "

    In addition to all the features Jinja supports, Copier includes:

    "}, {"location": "copier/#configuring-a-template", "title": "Configuring a template", "text": ""}, {"location": "copier/#the-copieryaml-file", "title": "The copier.yaml file", "text": "

    The copier.yml (or copier.yaml) file is found in the root of the template, and it is the main entrypoint for managing your template configuration.

    For each key found, Copier will prompt the user to fill or confirm the values before they become available to the project template.

    This copier.yml file:

    name_of_the_project: My awesome project\nnumber_of_eels: 1234\nyour_email: \"\"\n

    Will result in a questionary similar to:

    \ud83c\udfa4 name_of_the_project\n  My awesome project\n\ud83c\udfa4 number_of_eels (int)\n  1234\n\ud83c\udfa4 your_email\n

    Apart from the simplified format, as seen above, Copier supports a more advanced format to ask users for data. To use it, the value must be a dict.

    Supported keys:

    "}, {"location": "copier/#include-other-yaml-files", "title": "Include other YAML files", "text": "

    The copier.yml file supports multiple documents as well as using the !include tag to include settings and questions from other YAML files. This allows you to split up a larger copier.yml and enables you to reuse common partial sections from your templates. When multiple documents are used, care has to be taken with questions and settings that are defined in more than one document:

    You can use Git submodules to sanely include shared code into templates!

    ---\n# Copier will load all these files\n!include shared-conf/common.*.yml\n\n# These 3 lines split the several YAML documents\n---\n# These two documents include common questions for these kind of projects\n!include common-questions/web-app.yml\n---\n!include common-questions/python-project.yml\n---\n\n# Here you can specify any settings or questions specific for your template\n_skip_if_exists:\n    - .password.txt\ncustom_question: default answer\n

    that includes questions and settings from common-questions/python-project.yml

    version:\n    type: str\n    help: What is the version of your Python project?\n\n# Settings like `_skip_if_exists` are merged\n_skip_if_exists:\n    - \"pyproject.toml\"\n
    "}, {"location": "copier/#conditional-files-and-directories", "title": "Conditional files and directories", "text": "

    You can take advantage of the ability to template file and directory names to make them \"conditional\", i.e. to only generate them based on the answers given by a user.

    For example, you can ask users if they want to use pre-commit:

    use_precommit:\n    type: bool\n    default: false\n    help: Do you want to use pre-commit?\n

    And then, you can generate a .pre-commit-config.yaml file only if they answered \"yes\":

    \ud83d\udcc1 your_template\n\u251c\u2500\u2500 \ud83d\udcc4 copier.yml\n\u2514\u2500\u2500 \ud83d\udcc4 {% if use_precommit %}.pre-commit-config.yaml{% endif %}.jinja\n

    Note that the chosen template suffix must appear outside of the Jinja condition, otherwise the whole file won't be considered a template and will be copied as such in generated projects.

    You can even use the answers of questions with choices:

    ci:\n    type: str\n    help: What Continuous Integration service do you want to use?\n    choices:\n        GitHub CI: github\n        GitLab CI: gitlab\n    default: github\n
    \ud83d\udcc1 your_template\n\u251c\u2500\u2500 \ud83d\udcc4 copier.yml\n\u251c\u2500\u2500 \ud83d\udcc1 {% if ci == 'github' %}.github{% endif %}\n\u2502   \u2514\u2500\u2500 \ud83d\udcc1 workflows\n\u2502       \u2514\u2500\u2500 \ud83d\udcc4 ci.yml\n\u2514\u2500\u2500 \ud83d\udcc4 {% if ci == 'gitlab' %}.gitlab-ci.yml{% endif %}.jinja\n

    Contrary to files, directories must not end with the template suffix.

    "}, {"location": "copier/#generating-a-directory-structure", "title": "Generating a directory structure", "text": "

    You can use answers to generate file names as well as whole directory structures.

    package:\n    type: str\n    help: Package name\n
    \ud83d\udcc1 your_template\n\u251c\u2500\u2500 \ud83d\udcc4 copier.yml\n\u2514\u2500\u2500 \ud83d\udcc4 {{ package.replace('.', _copier_conf.sep) }}{{ _copier_conf.sep }}__main__.py.jinja\n

    If you answer your_package.cli.main Copier will generate this structure:

    \ud83d\udcc1 your_project\n\u2514\u2500\u2500 \ud83d\udcc1 your_package\n    \u2514\u2500\u2500 \ud83d\udcc1 cli\n        \u2514\u2500\u2500 \ud83d\udcc1 main\n            \u2514\u2500\u2500 \ud83d\udcc4 __main__.py\n

    You can either use any separator, like ., and replace it with _copier_conf.sep, like in the example above, or just use /.

    "}, {"location": "copier/#importing-jinja-templates-and-macros", "title": "Importing Jinja templates and macros", "text": "

    You can include templates and import macros to reduce code duplication. A common scenario is the derivation of new values from answers, e.g. computing the slug of a human-readable name:

    \ud83d\udcc1 your_template\n\u251c\u2500\u2500 \ud83d\udcc4 copier.yml\n\u2514\u2500\u2500 \ud83d\udcc4 name-slug.jinja\n

    It is also possible to include a template in a templated folder name

    \ud83d\udcc1 your_template\n\u251c\u2500\u2500 \ud83d\udcc4 copier.yml\n\u251c\u2500\u2500 \ud83d\udcc4 name-slug.jinja\n\u2514\u2500\u2500 \ud83d\udcc1 {% include 'name-slug.jinja' %}\n    \u2514\u2500\u2500 \ud83d\udcc4 __init__.py\n

    or in a templated file name

    \ud83d\udcc1 your_template\n\u251c\u2500\u2500 \ud83d\udcc4 copier.yml\n\u251c\u2500\u2500 \ud83d\udcc4 name-slug.jinja\n\u2514\u2500\u2500 \ud83d\udcc4 {% include 'name-slug.jinja' %}.py\n

    or in the templated content of a text file:

    # pyproject.toml.jinja\n\n[project]\nname = \"{% include 'name-slug.jinja' %}\"\n

    Similarly, a Jinja macro can be defined and imported, e.g. in copier.yml.

    slugify.jinja\n\n{# For simplicity ... -#}\n{% macro slugify(value) -%}\n{{ value|lower|replace(' ', '-') }}\n{%- endmacro %}\n
    # copier.yml\n\n_exclude:\n    - slugify\n\nname:\n    type: str\n    help: A nice human-readable name\n\nslug:\n    type: str\n    help: A slug of the name\n    default: \"{% from 'slugify.jinja' import slugify %}{{ slugify(name) }}\"\n

    or in a templated folder name, in a templated file name, or in the templated content of a text file.

    As the number of imported templates and macros grows, you may want to place them in a dedicated directory such as includes:

    \ud83d\udcc1 your_template\n\u251c\u2500\u2500 \ud83d\udcc4 copier.yml\n\u2514\u2500\u2500 \ud83d\udcc1 includes\n    \u251c\u2500\u2500 \ud83d\udcc4 name-slug.jinja\n    \u251c\u2500\u2500 \ud83d\udcc4 slugify.jinja\n    \u2514\u2500\u2500 \ud83d\udcc4 ...\n

    Then, make sure to exclude this folder in copier.yml

    _exclude:\n    - includes\n

    or use a subdirectory, e.g.:

    _subdirectory: template\n

    To import it you can use either:

    {% include pathjoin('includes', 'name-slug.jinja') %}\n

    or

    {% from pathjoin('includes', 'slugify.jinja') import slugify %}\n
    "}, {"location": "copier/#available-settings", "title": "Available settings", "text": "

    Remember that the key must be prefixed with an underscore if you use it in the copier.yml file.

    Check the source for a complete list of settings

    "}, {"location": "copier/#the-copieranswersyml-file", "title": "The .copier.answers.yml file", "text": "

    If the destination path exists and a .copier-answers.yml file is present there, it will be used to load the last user's answers to the questions made in the copier.yml file.

    This makes projects easier to update because when the user is asked, the default answers will be the last ones they used.

    The file must be called exactly {{ _copier_conf.answers_file }}.jinja in your template's root folder to allow applying multiple templates to the same subproject.

    The file must have this content:

    # Changes here will be overwritten by Copier; NEVER EDIT MANUALLY\n{{ _copier_answers|to_nice_yaml -}}\n
    "}, {"location": "copier/#apply-multiple-templates-to-the-same-subproject", "title": "Apply multiple templates to the same subproject", "text": "

    Imagine this scenario:

    All 3 templates are completely independent:

    You need to use a different answers file for each one. All of them contain a {{ _copier_conf.answers_file }}.jinja file as specified above. Then you apply all the templates to the same project:

    mkdir my-project\ncd my-project\ngit init\n# Apply framework template\ncopier copy -a .copier-answers.main.yml https://github.com/example-framework/framework-template.git .\ngit add .\ngit commit -m 'Start project based on framework template'\n# Apply pre-commit template\ncopier copy -a .copier-answers.pre-commit.yml https://gitlab.com/my-stuff/pre-commit-template.git .\ngit add .\npre-commit run -a  # Just in case \ud83d\ude09\ngit commit -am 'Apply pre-commit template'\n# Apply internal CI template\ncopier copy -a .copier-answers.ci.yml git@gitlab.example.com:my-company/ci-template.git .\ngit add .\ngit commit -m 'Apply internal CI template'\n

    Done!

    After a while, when templates get new releases, updates are handled separately for each template:

    copier update -a .copier-answers.main.yml\ncopier update -a .copier-answers.pre-commit.yml\ncopier update -a .copier-answers.ci.yml\n
    "}, {"location": "copier/#generating-a-template", "title": "Generating a template", "text": "

    You can generate a project from a template using the copier command-line tool:

    copier copy path/to/project/template path/to/destination\n

    Or within Python code:

    copier.run_copy(\"path/to/project/template\", \"path/to/destination\")\n

    The \"template\" parameter can be a local path, an URL, or a shortcut URL:

    If Copier doesn't detect your remote URL as a Git repository, make sure it starts with one of git+https://, git+ssh://, git@ or git://, or it ends with .git.

    Use the --data command-line argument or the data parameter of the copier.run_copy() function to pass whatever extra context you want to be available in the templates. The arguments can be any valid Python value, even a function.

    Use the --vcs-ref command-line argument to checkout a particular Git ref before generating the project.

    All the available options are described with the --help-all option.

    "}, {"location": "copier/#updating-a-project", "title": "Updating a project", "text": "

    The best way to update a project from its template is when all of these conditions are true:

    If that's your case, then just enter the destination folder, make sure git status shows it clean, and run:

    copier update\n

    This will read all available Git tags, will compare them using PEP 440, and will check out the latest one before updating. To update to the latest commit, add --vcs-ref=HEAD. You can use any other Git ref you want.

    When updating, Copier will do its best to respect your project evolution by using the answers you provided when copied last time. However, sometimes it's impossible for Copier to know what to do with a diff code hunk. In those cases, copier handles the conflict in one of two ways, controlled with the --conflict option:

    If the update results in conflicts, you should review those manually before committing.

    You probably don't want to lose important changes or to include merge conflicts in your Git history, but if you aren't careful, it's easy to make mistakes.

    That's why the recommended way to prevent these mistakes is to add a pre-commit (or equivalent) hook that forbids committing conflict files or markers. The recommended hook configuration depends on the conflict setting you use.

    Never update .copier-answers.yml manually!!!

    If you want to just reuse all previous answers use copier update --force.

    "}, {"location": "copier/#migration-across-copier-major-versions", "title": "Migration across Copier major versions", "text": "

    When there's a new major release of Copier (for example from Copier 5.x to 6.x), there are chances that there's something that changed. Maybe your template will not work as it did before.

    Copier needs to make a copy of the template in its old state with its old answers so it can actually produce a diff with the new state and answers and apply the smart update to the project. To overcome this situation you can:

    "}, {"location": "copier/#tasks-and-migrations", "title": "Tasks and migrations", "text": "

    tasks are commands to execute after generating or updating a project from your template. They run ordered, and with the $STAGE=task variable in their environment.

    # copier.yml\n\n_tasks:\n    # Strings get executed under system's default shell\n    - \"git init\"\n    - \"rm {{ name_of_the_project }}/README.md\"\n    # Arrays are executed without shell, saving you the work of escaping arguments\n    - [invoke, \"--search-root={{ _copier_conf.src_path }}\", after-copy]\n    # You are able to output the full conf to JSON, to be parsed by your script\n    - [invoke, end-process, \"--full-conf={{ _copier_conf|to_json }}\"]\n    # Your script can be run by the same Python environment used to run Copier\n    - [\"{{ _copier_python }}\", task.py]\n    # OS-specific task (supported values are \"linux\", \"macos\", \"windows\" and `None`)\n    - >-\n      {% if _copier_conf.os in ['linux', 'macos'] %}\n      rm {{ name_of_the_project }}/README.md\n      {% elif _copier_conf.os == 'windows' %}\n      Remove-Item {{ name_of_the_project }}/README.md\n      {% endif %}\n

    Note: the example assumes you use Invoke as your task manager. But it's just an example. The point is that we're showing how to build and call commands.

    Migrations are like tasks, but each item in the list is a dict with these keys:

    Migrations will run in the same order as declared in the file (so you could even run a migration for a higher version before running a migration for a lower version if the higher one is declared before and the update passes through both).

    They will only run when new version >= declared version > old version. And only when updating (not when copying for the 1st time).

    If the migrations definition contains Jinja code, it will be rendered with the same context as the rest of the template.

    Migration processes will receive these environment variables:

    # copier.yml\n\n_migrations:\n    - version: v1.0.0\n      before:\n          - rm ./old-folder\n      after:\n          # {{ _copier_conf.src_path }} points to the path where the template was\n          # cloned, so it can be helpful to run migration scripts stored there.\n          - invoke -r {{ _copier_conf.src_path }} -c migrations migrate $VERSION_CURRENT\n
    "}, {"location": "copier/#developing-a-copier-template", "title": "Developing a copier template", "text": ""}, {"location": "copier/#avoid-doing-commits-when-developing", "title": "Avoid doing commits when developing", "text": "

    While you're developing it's useful to see the changes before making a commit, to do so you can use copier copy -r HEAD ./src ./dst. Keep in mind that you won't be able to use copier update so the changes will be applied incrementally, not declaratively. So if you make a file in an old run that has been deleted in the source, it won't be removed in the destination. It's a good idea then to remove the destination directory often.

    "}, {"location": "copier/#apply-migrations-only-once", "title": "Apply migrations only once", "text": "

    Currently copier allows you to run two kind of commands:

    But there isn't yet a way to run a task only on the copy of a project. Until there is you can embed inside the generated project's Makefile an init target that runs the init script. The user will then need to:

    copier copy src dest\ncd dest\nmake init\n

    Not ideal but it can be a workaround until we have the pre-copy tasks.

    Another solution I thought of is to:

    That way instead of doing copier copy src dest you can do:

    copier copy -r 0.0.0 src dest\ncopier update\n

    It will run over all the migrations steps you make in the future. A way to tackle this is to eventually release a 1.0.0 and move the 0.1.0 migration script to 1.1.0 using copier copy -r 1.0.0 src dest.

    However, @pawamoy thinks that this can eventually backfire because all the versions of the template will not be backward compatible with 0.0.0. If they are now, they probably won't be in the future. This might be because of the template itself, or because of the extensions it uses, or because of the version of Copier it required at the time of each version release. So this can be OK for existing projects, but not when trying to generate new ones.

    "}, {"location": "copier/#create-your-own-jinja-extensions", "title": "Create your own jinja extensions", "text": "

    You can create your own jinja filters. For example creating an extensions.py file with the contents:

    import re\nimport subprocess\nimport unicodedata\nfrom datetime import date\n\nfrom jinja2.ext import Extension\n\n\ndef git_user_name(default: str) -> str:\n    return subprocess.getoutput(\"git config user.name\").strip() or default\n\n\ndef git_user_email(default: str) -> str:\n    return subprocess.getoutput(\"git config user.email\").strip() or default\n\n\ndef slugify(value, separator=\"-\"):\n    value = unicodedata.normalize(\"NFKD\", str(value)).encode(\"ascii\", \"ignore\").decode(\"ascii\")\n    value = re.sub(r\"[^\\w\\s-]\", \"\", value.lower())\n    return re.sub(r\"[-_\\s]+\", separator, value).strip(\"-_\")\n\n\nclass GitExtension(Extension):\n    def __init__(self, environment):\n        super().__init__(environment)\n        environment.filters[\"git_user_name\"] = git_user_name\n        environment.filters[\"git_user_email\"] = git_user_email\n\n\nclass SlugifyExtension(Extension):\n    def __init__(self, environment):\n        super().__init__(environment)\n        environment.filters[\"slugify\"] = slugify\n\n\nclass CurrentYearExtension(Extension):\n    def __init__(self, environment):\n        super().__init__(environment)\n        environment.globals[\"current_year\"] = date.today().year\n

    Then you can import it in your copier.yaml file:

    _jinja_extensions:\n    - copier_templates_extensions.TemplateExtensionLoader\n    - extensions.py:CurrentYearExtension\n    - extensions.py:GitExtension\n    - extensions.py:SlugifyExtension\n\n\nauthor_fullname:\n  type: str\n  help: Your full name\n  default: \"{{ 'Timoth\u00e9e Mazzucotelli' | git_user_name }}\"\n\nauthor_email:\n  type: str\n  help: Your email\n  default: \"{{ 'pawamoy@pm.me' | git_user_email }}\"\n\nrepository_name:\n  type: str\n  help: Your repository name\n  default: \"{{ project_name | slugify }}\"\n

    You'll need to install copier-templates-extensions, if you've installed copier with pipx you can:

    pipx inject copier copier-templates-extensions\n
    "}, {"location": "copier/#references", "title": "References", "text": ""}, {"location": "cpu/", "title": "CPU", "text": "

    A central processing unit or CPU, also known as the brain of the server, is the electronic circuitry that executes instructions comprising a computer program. The CPU performs basic arithmetic, logic, controlling, and input/output (I/O) operations specified by the instructions in the program.

    The main processor factors you should consider the most while buying a server are:

    "}, {"location": "cpu/#speed", "title": "Speed", "text": "

    To achieve the same speed you can play with the speed of a core and the number of cores.

    Having more cores and lesser clock speed has the next advantages:

    With the disadvantages that it offers lower single threaded performance.

    On the other hand, using fewer cores and higher clock speed gives the next advantages:

    And the next disadvantages

    "}, {"location": "cpu/#providers", "title": "Providers", "text": ""}, {"location": "cpu/#amd", "title": "AMD", "text": "

    The Ryzen family is broken down into four distinct branches:

    They're all great chips in their own ways, but some certainly offer more value than others, and for many, the most powerful chips will be complete overkill.

    "}, {"location": "cpu/#market-analysis", "title": "Market analysis", "text": "Property Ryzen 7 5800x Ryzen 5 5600x Ryzen 7 5700x Ryzen 5 5600G Cores 8 6 8 6 Threads 16 12 16 12 Clock 3.8 3.7 3.4 3.9 Socket AM4 AM4 AM4 AM4 PCI 4.0 4.0 4.0 3.0 Thermal Not included Wraith Stealth Not included Wraith Stealth Default TDP 105W 65W 65W 65W System Mem spec >= 3200 MHz >= 3200 MHz >= 3200 MHz >= 3200 MHz Mem type DDR4 DDR4 DDR4 DDR4 Price 315 232 279 179

    The data was extracted from AMD's official page.

    They all support the chosen RAM and the motherboard.

    I'm ruling out Ryzen 7 5800x because it's too expensive both on monetary and power consumption terms. Also ruling out Ryzen 5 5600G because it has comparatively bad properties.

    Between Ryzen 5 5600x and Ryzen 7 5700x, after checking these comparisons (1, 2) it looks like:

    I think that for 47$ it's work the increase on cores and theoretical RAM memory bandwidth. Therefore I'd go with the Ryzen 7 5700x.

    "}, {"location": "cpu/#cpu-coolers", "title": "CPU coolers", "text": "

    One of the most important decisions when building your PC, especially if you plan on overclocking, is choosing the best CPU cooler. The cooler is often a limiting factor to your overclocking potential, especially under sustained loads. Your cooler choice can also make a substantial difference in noise output. So buying a cooler that can handle your best CPU\u2019s thermal output/heat, (be it at stock settings or when overclocked) is critical to avoiding throttling and achieving your system\u2019s full potential, while keeping the whole system quiet.

    CPU Coolers come in dozens of shapes and sizes, but most fall into these categories:

    AIO or closed-loop coolers can be (but aren\u2019t always) quieter than air coolers, without requiring the complications of cutting and fitting custom tubes and maintaining coolant levels after setup. AIOs have also become increasingly resistant to leaks over the years, and are easier to install. But they require room for a radiator, so may require a larger case than some air coolers.

    Here\u2019s a quick comparison of some of the pros and cons of air and liquid cooling.

    Liquid Cooling Pros:

    Liquid Cooling Cons:

    Air Cooling Pros:

    Air Cooling Cons:

    "}, {"location": "cpu/#quick-shopping-tips", "title": "Quick shopping tips", "text": ""}, {"location": "cpu/#market-analysis_1", "title": "Market analysis", "text": "

    After a quick review I'm deciding between the Dark Rock 4 and the Enermax ETS-T50 Axe. The analysis is done after reading Tomshardware reviews (1, 2).

    They are equal in:

    The Enermax has the advantages:

    All in all the Enermax ETS-T50 Axe is a better one, but after checking the sizes, my case limit on the height of the CPU cooler is 160mm and the Enermax is 163mm... The Cooler Master Masterair ma610p has 166mm, so it's out of the question too. The Dark Rock 4 max height is 159mm. I don't know if I should bargain.

    To be in the safe side I'll go with the Dark Rock 4

    "}, {"location": "cpu/#ryzen-recommended-coolers", "title": "Ryzen recommended coolers", "text": ""}, {"location": "cpu/#cpu-thermal-paste", "title": "CPU Thermal paste", "text": "

    Thermal paste is designed to minimize microscopic air gaps and irregularities between the surface of the cooler and the CPU's IHS (integrated heat spreader), the piece of metal which is built into the top of the processor.

    Good thermal paste can have a profound impact on your performance, because it will allow your processor to transfer more of its waste heat to your cooler, keeping your processor running cool.

    Most pastes are comprised of ceramic or metallic materials suspended within a proprietary binder which allows for easy application and spread as well as simple cleanup.

    These thermal pastes can be electrically conductive or non-conductive, depending on their specific formula. Electrically conductive thermal pastes can carry current between two points, meaning that if the paste squeezes out onto other components, it can cause damage to motherboards and CPUs when you switch on the power. A single drop out of place can lead to a dead PC, so extra care is imperative.

    Liquid metal compounds are almost always electrically conductive, so while these compounds provide better performance than their paste counterparts, they require more focus and attention during application. They are very hard to remove if you get some in the wrong place, which would fry your system.

    In contrast, traditional thermal paste compounds are relatively simple for every experience level. Most, but not all, traditional pastes are electrically non-conductive.

    Most cpu coolers come with their own thermal paste, so check yours before buying another one.

    "}, {"location": "cpu/#market-analysis_2", "title": "Market analysis", "text": "Model ProlimaTech PK-3 Thermal Grizzly Kryonaut Cooler Master MasterGel Pro v2 Electrical conductive No No No Thermal Conductivity 11.2 W/mk 12.5 W/mk 9 W/mk Ease of Use 4.5 4.5 4.5 Relative Performance 4.0 4.0 3.5 Price per gram 6.22 9.48 2.57

    The best choice would be the ProlimaTech but the package sold are expensive because it has many grams.

    In my case, my cooler comes with the thermal paste so I'd start with that before spending 20$ more.

    "}, {"location": "cpu/#installation", "title": "Installation", "text": "

    When installing an AM4 CPU in the motherboard, rotate the CPU so that the small arrow on one of the corners of the chip matches the arrow on the corner of the motherboard socket.

    "}, {"location": "cpu/#references", "title": "References", "text": ""}, {"location": "css/", "title": "CSS", "text": "

    CSS stands for Cascading Style Sheets and is used to format the layout of a webpage.

    With CSS, you can control the color, font, the size of text, the spacing between elements, how elements are positioned and laid out, what background images or background colors are to be used, different displays for different devices and screen sizes, and much more!

    "}, {"location": "css/#using-css-in-html", "title": "Using CSS in HTML", "text": "

    CSS can be added to HTML documents in 3 ways: