Transwarp is a user-friendly compiler, that takes specification (in
the form of *.stf files) as input, and helps you generate code,
documentation, test cases, or whatever you like! Your documentation
will never be outdated again!
You can use stf files to describe protocols, data format, parsers,
or pretty much anything that deals with structured data. The output
can be source code, documentation, or any other user- or
machine-readable thing you write a template for.
The input data format is simple, and very easily extensible. It only enforces the most basic structure - you decide the rest!
+--------------------------+ +-------------------------+
| | Templates can load plugins | |
| Template files (*.tpl) <--------------------------------+ Compiler plugins (*.py) |
| | | |
+-+------------------------+ +-------------------------+
|
| Compiler loads templates +----------------------+
+---------------------------> | Compiler writes output
| Transwarp Compiler +-------------------------+
+---------------------------> | |
| Compiler loads stf files +----------------------+ |
| |
+-+------------------------+ +------------------v------+
| | | |
| Source files (*.stf) | | Generated content |
| | | |
+--------------------------+ +-------------------------+
To use transwarp, you need a specification, and a corresponding template (or set of templates). The transwarp compiler parses all input files, and generates the output from your templates.
After all input files are parsed, the data is put into a tree structure, which is made available for the template code to inspect and transform. Since the templates are completely in charge of the output, there are no requirements for how (or how much) you have to use the available data. More importantly, you can start by making your existing source files into templates, and slowly build conversion logic into them!
The widely used, well-tested and well-documented Mako template system is used for templating.
Okay, so you want to try out transwarp! Let's take a look at generating parser code and templates for the Artemis Space Bridge Simulator game.
Here's what you need:
- python 3
- The transwarp compiler (https://github.com/chrivers/transwarp)
- A protocol specification (for example, https://github.com/chrivers/isolinear-chips)
- Templates for that protocol (for example, https://github.com/chrivers/duranium-templates)
# download and install transwarp
$ git clone https://github.com/chrivers/transwarp
$ cd transwarp
$ ./setup.py -q install # use sudo if you want to install system-wide
$ cd ..
# as an example, get artemis protocol spec
$ git clone https://github.com/chrivers/isolinear-chips isolinear
# as an example, get rust code templates
$ git clone https://github.com/chrivers/duranium-templates duranium
# to see the complete set of options:
$ transwarp --helpIf you get a list of commend line arguments, transwarp is installed correctly. Otherwise, please open an issue so we can fix the problem.
Now we are ready to compile!
# let's see a summary of what transwarp would like to compile
$ transwarp -D isolinear/protocol -I duranium/templates -L duranium/lib -O output-dir -s
[*] Will create:
output-dir/client/mod.rs
output-dir/client/reader.rs
output-dir/client/writer.rs
output-dir/enums.rs
output-dir/maps.rs
output-dir/server/mod.rs
output-dir/server/object.rs
output-dir/server/reader.rs
output-dir/server/update.rs
output-dir/server/writer.rs
output-dir/structs.rs
...Let's do a quick breakdown of the arguments here:
-D isolinear-chips/protocol
This points transwarp to the protocol specification. From this
directory, all *.stf files are loaded. The files are always loaded
in alphabetical order, for consistency between runs. The templates
will have access to each file, as a variable with the filename
(without extension), prefixed by underscore.
Example: protocol.stf, will be available as _protocol in the
templates.
-I duranium-templates/templates
Here we point transwarp to the templates. This directory is
recursively scanned for templates, to be generated. The default
template file extension is .tpl, but this can be changed with the
--extension (-e) option. For each input template, an output file will
be generated in the output dir, except for the .tpl extension.
Example: server/writer.rs.tpl will be output to server/writer.rs.
-L duranium-templates/lib
Templates can load python code into the templates, which advanced templates can use to do more serious data processing. The duranium templates (which generate rust code), use a single such plugin module. This is not strictly required, but makes the template code cleaner, since some repeated functionality can easily be shared in one place.
Simpler templates, such as documentation generators, wouldn't have to use any plugins at all. It is very reasonable to write self-contained templates.
-O output-dir
Here we set the output directory. The relative path of the input file from the input-dir is used as the output file name.
-s (summarize)
Transwarp defaults to showing a diff (like git diff), but since the
files have not been initially generated yet, this would produce a very
large amount of output. To avoid this, we use -s to produce only a
list of changes.
If we agree with the summary, we can write all the files by appending
-u (update mode):
$ transwarp -D isolinear/protocol -I duranium/templates -L duranium/lib -O output-dir -u
[*] Updated output-dir/server/mod.rs
[*] Updated output-dir/structs.rs
[*] Updated output-dir/server/object.rs
[*] Updated output-dir/server/writer.rs
[*] Updated output-dir/client/writer.rs
[*] Updated output-dir/client/reader.rs
...
Transwarp looks at the modification time of both stf files and template files, and tries to only update changed files, so if we run it again, we can see we are synchronized:
$ transwarp -D isolinear/protocol -I duranium/templates -L duranium/lib -O output-dir -u
[*] All templates up-to-date
Render templates from "foo" into "bar":
$ transwarp -I foo -O bar -u
Summarize changes caused by rendering templates from "src" into current dir:
$ transwarp -D someproto -I src -s
Show diff caused by rendering templates from current dir (recursively) into "target":
$ transwarp -D myspec -O target
Enable verbose mode to explain what is going on behind the scenes:
$ transwarp -D data -I template_dir -O output_dir -v
Enable quiet mode to compile from build scripts without unnecessary output:
$ transwarp -D specfiles -I ../templates -O src/protocol-parser/ -q
The input data format, the Simple Type Framework (.stf), is a user-friendly text-based format, designed to be easily human-readable while still being parsable.
There are no keywords in stf, only a few different types of structure. The stf files describe a data structure, which is then used by the template in whatever manner they wish. This leaves you in charge of how to structure and organize your specification data.
Any line that has ## as the first non-whitespace contents, is
regarded as a comment line, and will be completely ignored.
There are exactly 4 kinds of "things" in stf files:
- blocks
- fields
- types
- constants
There are no reserved words, and no identifiers have special meanings. Let's discuss each of them in more detail.
A constant is simple to define. The syntax is the following:
name = 0x{hexvalue}
The name can consist of all alphanumeric characters, and the integer must be in hex format. Here's an example of a list of constants:
InfusionPCoils = 0x00
HydrogenRam = 0x01
TauronFocusers = 0x02
CarapactionCoils = 0x03
PolyphasicCapacitors = 0x04
CetrociteCrystals = 0x05That's all you need to know about constants.
If you need to store something that isn't an integer constant, you'll want to use a field. The syntax is:
name : type
Notice that we use : instead of =. That's what separates fields
from constants! In the next section, we will take a look at how types
work, but here are some examples of fields for now:
index: u32
vessel_type_id: u32
x_coordinate: f32
y_coordinate: f32
z_coordinate: f32
pitch: f32As you can probably surmise, "u32" and "f32" are valid types. But remember, there are no built-in types, keywords, or reserved identifiers in stf! In fact, all simple strings are valid type names. It is up to you (in the templates) to give these names meaning!
But types can do more, when you need them to. We'll take a closer look in the next section.
As we saw in the previous section, any string of alphanumeric characters is a valid type: However, types have a powerful feature: type arguments. Any type can have type argument, and they are written in angle brackets, like so:
type<arg1, arg2, ...>
All type arguments are, themselves, types! This means that your type arguments can have type arguments, which can have.. you get the idea. Types are really just a compact way to describe a tree structure with named nodes.
If a type is not followed by angle brackets, it has 0 type arguments. Here are some examples of valid types:
f32
u32
string
GameMasterButton<u8>
DriveType<u8>
map<ShipSystem, f32>
map<TubeIndex, TubeStatus<u8>>
Again, none of these examples assume anything about the names given,
and it is completely up to you to pick the names you would like to
use. It is syntactically valid to construct a type called
u32<bool<string<with_milk>>>, but it might be harder to come up with
a reasonable explanation for what it means. But that's your job ;-)
Now that we have the basic kinds of content explained, let's see how we can group it together to make it more useful.
A block is, in essence, a namespace for a group of things. Think of it as a container with a name you can refer to.
Here's an example of the block syntax:
expr name
const1 = 0x01
const2 = 0x02
const3 = 0x03
...
field1: type1
field2: type2<foo>
field3: type3<a, b, c>
...There's a few things going on here, so let's take a look at them one at a time.
First of all, you start a block by writing 2 names on a line, with any
amount of whitespace in between. The name part is used to refer to
the block from the templates, and expr is any kind of descriptive
word you like, to give your templates something some more
information. For instance, you might like to refer to a block as an
enum, a struct a class, or something like that. But you could
just as well have a document, a protocol or a spaceship.
Second, we can see that blocks contain any number of constants and fields. These follow the exact syntax described earlier, so there should be no surprises here. It is customary (but not enforced) to put constants first, and fields later, although having blocks with both constants and fields is a uncommon in practice.
Third, you need to put exactly 4 spaces in front of all content that goes in a block, to mark it as being part of that block. This is a little rigid, and could maybe be relaxed in a future stf specification, but for now it's always 4 spaces.
Finally, you should know that only the header (expr + name) is mandatory - blocks do not have to have any constants or fields at all. Also, you can put blocks inside blocks (.. inside blocks, etc). As you have probably guessed by now, stf is not going to tell you how to structure your data. Of course, you add 4 more spaces for each additional level of nesting.
That was a bit of a mouthful, so let's take a look at some valid blocks:
enum MediaCommand
Delete = 0x00
Play = 0x01
Create = 0x02Here we have a block with the name MediaCommand and the expr
enum. It contains 3 constants, named Delete, Play and Create.
Let's see a slightly more complex example:
struct PlayerShipUpgrades
mask_bytes = 11
active: map<UpgradeType, bool8>
count: map<UpgradeType, i8>
time: map<UpgradeType, u16>This is a block with the name PlayerShipUpgrades and expr
struct. It contains 1 constant, mask_bytes and 3 fields, active,
count and time.
Finally, let's take a look at an example of nested blocks:
packet ServerPacket
struct AllShipSettings
ships: map<ShipIndex, ShipV240>
struct BeamFired
id: i32In this example, our ServerPacket block contains 2 blocks, named
AllShipSettings (which contains the field ships) and BeamFired
(which contains the field id), respectively.
Now that you know how blocks, fields, types and constants work, you are free to combine them in any way you like!
There are a few advanced features available, designed to make the format easier and more powerful to work with. Let's go through all three of them here, and afterwards you will know everything there is to know about the format!
Making comments in the stf files is not a bad idea, but what if you'd
like to use access your descriptions and examples from inside the
templates? No problem, you simply need to mark a comment with # (doc
comment) instead of ## (regular comment). Lines marked when # will
NOT be ignored by the compiler, but will instead have their text
contents appended to the next item. Here are some examples:
## this line is completely ignored by the compiler
## the next line is a doc comment (but this line is not)
# this enum is super important, because reasons
enum VeryImportant
Wow = 0x10
NotWow = 0x20
# this struct is less important than the enum
struct LessImportant
## You can also add docs for constants and fields, of course!
## this is demonstrated below
# compared to VeryImportant, this is definitely not as good
KindOfWow: bool
# remember to do that thing here, because of...
## you can inject normal comments in the middle of doc comments too
## and it will work as expected (be ignored)
# ..the reasons for doing it
NoBigDeal: u32To summarize: Like normal comments, doc comments do nothing on their own, but unlike normal comments, they are available to the templates, meaning you can use them to provide descriptions for your items, for example by converting them into comments in whatever target language your templates generate.
This project is licences under the GNU General Public Licence (GPL) Version 3.
In the Star Trek universe, Transwarp refers to speeds faster than Warp 10. This is the theoretical warp speed limit, much the like the speed of light is limit in our universe.