Development Diary #001: Long time, no see!

[StjerneIdioten]

Hi there!

It's been a while since I did one of these. As a matter of fact it's been a while since I did much of anything with respect to the exporter 😉 I have been popping my head into the various support channels from time to time, but that has been just about as much work as I have put into the exporter for the last 3 months due to my studies taking precedence. I am hopefully now at a point where I can show it some love again and boy do I have plans!

The Infamous Rework

Although I haven't put in any frontfacing work for a while, I have pondered over how to crack this thing open and solve the problems I presented in my previous Dev Diary. In the last few days I've even started to implement and test some of my ideas as development should be an interative process of constant improvement 😉

GE-Core

I talked about splitting the "core" python code from the integrations with the Blender UI to make not only testing easier but also to decouple the code a lot more. This combined with my wish to create certain other tools that doesn't necessarily have anything to do with exporting from Blender to the I3D format has led me to completely split the "core" code off from the Blender code. It will from here on out be it's own project called "GE-Core" and will be developed in its own github repository (that I am currently keeping private until it has a little bit more meat on it) the plan is for it to be it's own standalone library for manipulating anything to do with Giants Engine modding. This also means that I will be setting up the repository for GE-Core to automatically build, unit test and upload to PyPi (the Python Package Index) for others to integrate into any project where they might require to interface with any of the GE modfiles. I hope this decoupling will make the code first and foremost more robust but also easier to change and update as new FS games come out, with new features.

I have worked a little bit on the architecture for the project and implemented some parts of it. But here in the beginning it will be going a little bit slower as I have to setup all of the development specific stuff as well such as unit tests and the build servers. I will explain further about the different parts in the next sections, but the gist of it is to keep everything as modular as possible so you can swap between different import/export methods and "FS standards" which is just my fancy way of saying that I hope to support multiple FS versions with this one release as some people tend to stay back a few versions or maybe want to import some old mod and rework it for the newer generations. This whole thing should work by an interface (basically an agreement about functions used) that both import and export implementations support and in theory anyone should be able to come in and write a custom importer/exporter if they so wish as long as they stick to the interface.

(The image can be clicked for a higher resolution version)

Unit Testing

Unit tests, unit tests and more unit tests. I keep mentioning them, but why? Basically while developing the software bugs will sneak their way in, they are pesky little things (that cause big problems from time to time) and I might not think of everything either when implementing something. To combat this you can write unit tests, which are tests that you run on "units" of your source code. This also forces you to write your software in manageable chunks that can actually be tested thus reducing coupling (at least in theory)

The good thing is that these test suits are often written in a way where they can be automatically run by your build system, thus you can write a series of tests for your system and run them any time you update anything (or a library you depend on is updated) and immediatly know if some change is now causing problems. Of course the testing is only as good and dependable as the tests you write, but in general this should make it more safe to update things.

Specifically in this project I will be using pytest which isn't the testing library that comes with python natively, but from what I've seen it looks easier to use and with better results also since this is used during the development part of the project, I can use any package I want since I don't have to consider how to get it into Blender's internal python version.

Another metric that I can put into the project now is code coverage which is basically just a measure for how much of your source code is actually tested by your unit tests, thus how "safe" you are in case of change.

Blender Exporter

Hurray! Blender has finally decided to not stick with being compatible with the vfxplatform spec and has included python 3.9.2 for their 2.93 release! This allows me to do a few on the development side (Especially use the wallrus operator!) and not rely on older libraries.

For the Blender Exporter itself I will treat it as a UI that I build up around the new GE-Core library functionality thus keeping all the UI stuff completely separated and for itself. I will stick with this repository, but move relevant issues to the GE-Core library at some point.

Speed wise I hope this change in architecture wont affect things to much. A lot of the access to data can be implemented smartly with generator functions in python and data access is kind of slow in Blender anyway as is, since they are more focused on keeping the footprint of the .blend files down (Thus most things are looked up every time they are accessed, such as child/parent relationships)

I am hoping to have some form of unit testing for the Blender integration as well, but it is a little bit harder to do since we are talking about a UI and that isn't as easy to unit test. Most things in the UI are called through underlying operator functions anyway though and the entire UI is accesible without actually interacting with things through a cursor/keyboard. But in all depends on how Blender works in headless mode. I at least hope to catch any obvious things like UI code that won't even load due to some data structure change between Blender versions. There is a pytest plugin available for this called pytest-blender that looks promising and I hope I can use it to run the exporter against several different Blender versions as well, to further maintain compatibility as Blender goes through updates. I haven't planned how to handle any major structural changes that might happen yet, but I will consider that once it becomes a thing to consider. Right now the project isn't mature enough anyway for it to be a thing and it will be easier to thing about once I have some more to work with.

Managing Dependency Versions

To summarize a bit I hope to be able to maintain integration with different Blender versions, instead of the current model of just jumping to the next Blender version and leaving the old legacy version as is without updates.

I also hope to have compatability with different versions of FS and the formats used as Giants decides to update them.

Hopefully a lot of this will end up running through more or less automated processes, at least for the checking part. And I would only have to lift a finger on behalf of compatability when there is an actual incompatability (But now I should be able to at least guarantee that there is none with 99% confidence)

Timeline

All of these grandiose plans are of course mostly plans at this point, with a bit of code and gruntwork sprinkled in. As I've mentioned previously I will update the existing exporter if there are som seriously bad bugs that pops up, but I will mostly focus on getting this restructuring on its way. Since there is release date for FS 22 now, this is the "deadline" I will work towards for having the new exporter ready. Such that people can use it for updating their existing mods and creating new mods for the game. At the moment none of us knows anything about the changes to come, so the existing exporter will be good enough for creating mods for now.

Most of the actual code that I have to write is already present in the existing exporter, thus once I've setup the new structure I will be doing a lot of "copy/pasting" to say. Mostly it will be fitting the old code into the new system, but I don't have to figure out how things actuallly work as I go, like I did the first time around.

This is of course tangiable at best and comes with the caveat that I cannot see into the future and I still have my studies to tend to as well, but I hope that I can make everything fit and keep this project in the loop. Most of the seriously uncertain things about my studies have been resolved in the last few months, thus making things a bit more stable.

Next Time

For my next developer diary I will hopefully have a little bit more to show for the GE-Core library, maybe go over the build system a little bit and hopefully have publicated the repository. As always I am looking forward to try my best at making the best possible exporter for you guys 😉