[Enhancement]: Improve the workflow for local development of Pretendo

[ashquarky]

Update: Plan of action here #57 (comment)

Checked Existing

• I have checked the repository for duplicate issues.

What enhancement would you like to see?

When developing the Pretendo Network itself, Inkay can serve as a roadblock since it makes assumptions about the production Pretendo Network - domains, certificates, etc.

This is a tracking issue to discuss specifically how Inkay can better serve developers of Pretendo Network. We do not intend to deal with applications outside of local development hosting at this time.

Any other details to share? (OPTIONAL)

Parts of this have been discussed before: #40 (comment) #39

I think it's okay to assume developers will use a proxy, but there are still issues from having to write all the domain rewrite rules in one's preferred proxy suite (Fiddler, Charles and mitmproxy are all in common use) and the proxy's self-signed certificates.

We also need to determine if we should assume developers will use a full self-hosted setup, or a "hybrid" setup (e.g. production Accounts/Friends but development Juxt/NEX). I think there is potential in a hybrid setup combined with e.g. a mode on the Juxt backend that doesn't validate account tokens, or a mode of the NEX servers that read passwords from a nex-viewer-format config file instead of querying Accounts.

@CaramelKat
@TraceEntertains
@jonbarrow (optional)

[jonbarrow]

I agree there is virtue in trying to improve the developer experience. However, I still personally feel like this is out of scope for Inkay, and that it should be kept as simple as possible and within the scope of "production patches" for us. If people need to work locally, the existing options are pretty trivial:

• Build Inkay from source (minor changes, would be better if we had explicit build instructions)
• Use a proxy server (trivial to do and is honestly the fastest way to get this going, it's how I did all of my development prior to Inkay and is how I still do things)

I understand your point on the number of rewrite rules, but really there isn't that many. Fiddler, Charles, and mitmproxy all support wild card style rewrites, so you can just do something akin to https://*.nintendo.net -> http://*.pretendo.cc (or whatever domain is being used) and for 99% of the domains it will Just Work:tm:

As for the certificates, this has really never been an issue since SSL patches have existed for a long time. In fact, we have several of them. Our org has https://github.com/PretendoNetwork/wiiu-nossl which is private for some reason (not sure why it's private? But I do give out builds of this to people at times to help with debugging, so it's a useful tool to have at our disposal), and then there's the other private repo you and Rambo have added stuff to

Opening these SSL patches to the public as (a) dedicated development tool(s) + using a proxy server with some basic rewrite rules would be a better way to go imo rather than trying to mix the scopes of "development tool" and "production user tool" into Inkay

Alternatively, part of the reason why @MatthewL246 was brought on was because of his Docker work with the idea of creating a dedicated development environment. That, also, sounds like a better plan imo than trying to add development stuff to Inkay, and https://github.com/PretendoNetwork/mitmproxy-nintendo already serves the purpose of directing Nintendo requests to a custom server which can easily be modified to use whatever domain (or do what I do and just change your hosts file)

We also need to determine if we should assume developers will use a full self-hosted setup, or a "hybrid" setup (e.g. production Accounts/Friends but development Juxt/NEX)

While there is sometimes virtue in a hybrid setup (and is, to be honest, what I use most of the time, though I'm getting myself out of that habit) I think officially we should encourage full self-hosted setups. A development environment really shouldn't be mixing with production imo, even if it is arguably more convenient. That way we reduce the risk of corrupting live production data, and the development environment is not tied to whatever is in production (as much as there is virtue in using a hybrid setup, there is just as much, if not more, virtue in having everything isolated and fully within the developer's control)

Also, we should consider moving some of these instructions (setting up the local servers, installing the languages/runtimes, getting your patches ready and compiled, etc.) to https://developer.pretendo.network (repo https://github.com/PretendoNetwork/developer-documentation), since this supposed to be the main source of information for developers eventually (a sister issue is probably in order on this repo? Though there are these 3 issues already PretendoNetwork/developer-documentation#5, PretendoNetwork/developer-documentation#6, and PretendoNetwork/developer-documentation#7 which all somewhat serve this goal)

[ashquarky]

Build Inkay from source (minor changes, would be better if we had explicit build instructions)
Use a proxy server (trivial to do and is honestly the fastest way to get this going, it's how I did all of my development prior to Inkay and is how I still do things)

I think these are both reasonable things to request developers do, but I still think streamlining is possible even for these workflows. For example, could the dockerised development environment automatically provide a suitably patched Inkay, or expose a DNS/proxy server with the proper redirections applied?

Build Inkay from source (minor changes, would be better if we had explicit build instructions)

Inkay's URLs are currently spread across several files:

• Accounts, Friends, NEX, SpotPass
• Miiverse applet (trust me, there's a URL in those bytes!)
• Miiverse API
• etc.. (Account Settings, eShop, and NNCS all have their own files with URLs)

I think it would be valuable to factor out the pretendo.cc part into one location in the root of the repository so that it can be replaced with pretendo.localhost or whatever easily or even automatically. This would not affect how production builds of Inkay operate at runtime - this is just a build-time option.

Use a proxy server (trivial to do and is honestly the fastest way to get this going, it's how I did all of my development prior to Inkay and is how I still do things)

Using a proxy means self-signed certificates, and the NoSSL plugins that exist do not reach into web applets (Miiverse, eShop, Account Settings) and thus do not help with development of these services.

Either we could refactor Inkay's handling of the production SSL cert (as has been discussed in the past), or extend one of the NoSSL plugins to work for the applets, now that Aroma has improved enough to make that possible. I'm okay with either approach - assuming you want the latter, I think we should transfer the Meowth repository to the Pretendo org and make it into a proper dev tool.

So, the setup would be something like:

• main Aroma environment
  • Prod Inkay
• test Aroma environment
  • Locally built Inkay with different domains (if needed for the individual devs' proxy setup)
  • NoSSL plugin (updated to cover all development targets)

While there is sometimes virtue in a hybrid setup (and is, to be honest, what I use most of the time, though I'm getting myself out of that habit) I think officially we should encourage full self-hosted setups.

I think we can agree to disagree on this - specifically services like Juxt are practically completely independent of the account server and should be hostable standalone. It also says a lot that as far as I know ~all of the Pretendo team are using hybrid setups on prod Accounts since the Docker stuff (as far as I'm aware) isn't actually usable yet.

Maybe a mock Accounts server that just responds with "yeah sure lol" to every request instead of requiring an actual registration, linking, email, and seperate Wii U profile? Not 100% sure, maybe we can discuss this further in another issue.

[ashquarky]

Oh and I had honestly forgotten about the developer wiki, will mull on it and read up!

[jonbarrow]

I think these are both reasonable things to request developers do, but I still think streamlining is possible even for these workflows. For example, could the dockerised development environment automatically provide a suitably patched Inkay, or expose a DNS/proxy server with the proper redirections applied?

It could, and this sounds reasonable to me. Though I still disagree with providing a custom build of Inkay in this case since it feels like it still makes it "officially" a development tool on our part, whereas I believe the scope should stay limited to just production patches (I have more on this further down)

Inkay's URLs are currently spread across several files
...
I think it would be valuable to factor out the pretendo.cc part into one location in the root of the repository so that it can be replaced with pretendo.localhost or whatever easily or even automatically. This would not affect how production builds of Inkay operate at runtime - this is just a build-time option.

Oof yeah I didn't realize just how spread it was (and I forgot about the bytes one). Valid! I agree this should be consolidated, that's a good area for this to be improved on

Using a proxy means self-signed certificates, and the NoSSL plugins that exist do not reach into web applets (Miiverse, eShop, Account Settings) and thus do not help with development of these services

For this, see my comment here PretendoNetwork/Martini#13. These applets are difficult to work with in general, even if you get the certificates part figured out. Using the Wii U as the development platform is not great and is, honestly, probably the worst way to go about developing for the applets ironically. They're slow, annoying to do basic things like refreshing, etc.

I think a better solution to this is to move that task to Cemu. Whether that be through us providing a way to load these applets in Cemu (like with Martini-style patched RPXes/RPLs) or by maintaining a dedicated fork of Cemu specifically for working in the applets (which would be more difficult on our part, but would give much more control over the workflow) and for anything not using an applet then the expectation is to either build Inkay from source or use a proxy server + an SSL patch

I also joked with @CaramelKat a few years ago about building a custom browser which uses a super old version of WebKit to try and emulate the applet environment, which would honestly be the best solution (maximum control, no reliance on console emulation, etc.), but is also very much the most unrealistic (build a dedicated browser? Even if we piggy-backed off existing tech like WebKit that's still a ton of work)

I think we should transfer the Meowth repository to the Pretendo org and make it into a proper dev tool

That sounds great to me 👍

I think we can agree to disagree on this

I think I may have misunderstood you originally. When you said "hybrid" and talked about only hosting parts of the network, I thought you were implying that there should be a mix of "this uses a local server, but this uses a production server". Your follow up now reads more like "Juxtaposition doesn't rely on NEX, and NEX doesn't rely on Juxtaposition, so those things don't necessarily need to be hosted at the same time". The latter I am in agreement with, things which don't necessarily rely on each other don't necessarily need to be hosted together either. I just figure it would be easier to spin everything up at once for the sake of simplicity. If you did mean "use local data in some places and production data in others", then I do still disagree with that and there's a number of times where that breaks things. For example, SMM stores a list of your uploaded courses in your account save file. If you try to use a production account on a local server, then the course data won't match and SMM will think your courses have been deleted on the server and mark them as such in your save file. You would need to make a fresh account to avoid this issue, and at that point just run everything locally and don't take up an account on production for that. There's also plenty of times where you would explicitly want everything locally, for example if the account server was changing the encrypted token format, you'd want to host the account server locally with those changes (which may not be in prod yet) and update another server alongside them. This introduces the need to sometimes use production data, sometimes use local data, which overall just complicates the development setup imo. I still also very much have concerns about a developer working locally accidentally fucking prod (the last thing we need is a "oops I dropped the prod database" issue at 3am, looking at you @mrjvs @binaryoverload). That just sounds like a recipe for disaster, and I think it's well worth the minor annoyance of running an extra server or two to avoid that

[jonbarrow]

Oh and I had honestly forgotten about the developer wiki, will mull on it and read up!

It's super incomplete. It's mostly on hiatus (at least the NEX parts) while we finalize our library designs. Ofc other non-NEX lib additions can still be made in the meantime

[CaramelKat]

Forgive me if this is a little rough, this is currently getting typed out on my phone at 9% lol

Regarding the reliance on other servers, Juxt is unfortunately no where near as independent as it once was. Once upon a time all of the account data was mirrored. This offered a lot of flexibility in terms of speed and how we processed the data, but had risk for getting out of sync. For the last few years, this data has been pulled directly from the account server (previously through a db connection, now via grpc and cached locally for a given session for read only operations since it's slow to get the data).

This does mean a self hosted account server is almost certainly necessary. I've avoided this for years using a prod environment, but had to cave a few months back when I tried to get back into development (and promptly gave up because I couldn't get it or the friends sever running after nearly a week of effort, even with the newer docker infrastructure). Assuming the docker setup actually works now (again, haven't tried in months) having it easier to build with local certs and domains would be a huge boost to development. I've learned the hard way that there isn't a better way to test changes in Juxt than on console, especially on the 3DS.

Cemu would also be a good solution, and I'd love to explore that as a development avenue as well

[jonbarrow]

The 3DS is good to mention, we had been ignoring that in this conversation. I still think local applet development would be best done in emulators, seeing as you get all the benefits of not running it on native hardware (faster to load, faster to iterate on, potential for making a dedicated environment through forking/stripping, etc.)

Maybe Azahar would be a good candidate for this? Or we can assume that anything the Wii U applet can run, the 3DS applet can as well (as in, functionally speaking. Ofc the 3DS has more limited resources and so things like asset sizes and all that will be different, I mean in terms of like HTML, CSS and JS support) and just use Cemu to work on both platforms?

[ashquarky]

Cemu isn't a low-level emulator and thus won't be perfectly accurate, so I think we should always have some ability to test on hardware too, even if it's just as a pre-commit smoketest.

I think just supporting everything is reasonable here and developers can pick whichever workflow works best for them.

[ashquarky]

This does mean a self hosted account server is almost certainly necessary.

I'll set aside the hybrid idea for now then and focus on full selfhosting.

[jonbarrow]

Cemu isn't a low-level emulator and thus won't be perfectly accurate, so I think we should always have some ability to test on hardware too, even if it's just as a pre-commit smoketest.

I think just supporting everything is reasonable here and developers can pick whichever workflow works best for them.

I'm not saying we should strictly disallow hardware testing, and from my tests Cemu has been perfectly fine when it comes to applets (the issue with some graphics being wonky is a little unfair on my part since the system I tested this on was barely able to run Cemu at all, it had a myriad of other issues with normal titles as well. Anyone using anything even remotely modern should have a much better experience than I), but I do think it should be our recommended "go-to" method given the benefits it brings and the way it simplifies the developer environment, and we have yet to see any real negative results from using Cemu for this outside of it having some issues on potato machines like mine (which we imo should be considered an outlier, my machine really doesn't have to be considered typical imo). Anything else (building Inkay, running on real hardware, etc.) imo should be considered more of a "you can, but you'd likely have a worse time" sort of deal

Though you do make a good point about doing pre-commit smoketests. Checking on real hardware before committing should probably be mandatory (I've only been talking about how things would be setup for active development)

The more we talk about this the more I want to just build our own dedicated applet browser emulator (/s)

The applets suck

[jonbarrow]

@ashquarky On a more serious note, you mentioned Cemu not being low-level. What about Decaf? I haven't checked up on that in a few years, but surely it's gotten better no? And iirc it was low-level?

There's also https://github.com/kinnay/Wii-U-Firmware-Emulator which specifically says:

This emulator emulates the Wii U processors and hardware at the lowest level. It's currently able to emulate all the way through boot1, IOSU and Cafe OS up to the Wii U menu

though it's nowhere near as complete as something like Cemu. But maybe it would be a better candidate for a dedicated applet tester outside of real hardware?

Just spit-balling ideas here

[ashquarky]

I think Cemu is generally better than Decaf these days for accuracy and compatibility, and while Decaf does go a bit harder on things like IOS HLE it's still ultimately a still HLE emulator and those inherently have some amount of inaccuracy or bugs.

WUFE is indeed a perfect low-level emulator, but it also doesn't have a graphics output, so stones glass houses etc.

To be clear I think Cemu will be an excellent DX speedup for the vast majority of changes, I just also think that since our target is hardware we should test on hardware too. Plus hardware is arguably easier for new contributors, since they already have the application and account files installed and ready to go, so it's a bit less setup.

[jonbarrow]

I think Cemu is generally better than Decaf these days for accuracy and compatibility, and while Decaf does go a bit harder on things like IOS HLE it's still ultimately a still HLE emulator and those inherently have some amount of inaccuracy or bugs

Damn, I was mistaken then. I thought Decaf was LLE

WUFE is indeed a perfect low-level emulator, but it also doesn't have a graphics output, so stones glass houses etc.

Damn again, forgot there was no graphics output. I mean. I guess that CAN be added, but that does place a lot more work on us (maybe this should still be added to the tasklist however? As a "in case anyone wants to go for it" thing?)

To be clear I think Cemu will be an excellent DX speedup for the vast majority of changes, I just also think that since our target is hardware we should test on hardware too

Yeah this is fair. I think we agree there. Imo the ideal workflow would be:

• Develop against Cemu, as a way to speed things up since it's much faster to iterate on
• Test changes on real hardware before committing, to confirm changes work as expected

Plus hardware is arguably easier for new contributors, since they already have the application and account files installed and ready to go, so it's a bit less setup

I think we should have some reasonable expectations of people who want to contribute in ways that require a development setup, personally. For things like minor website changes, really anyone can do that. But if you want to contribute to the core parts of the network, I think it's fine if we have some higher expectations and ask them to do things like "use Cemu", which really isn't that tall of an order to ask tbf. If you're wanting to contribute like that, you're already setting up several things on your PC. Expecting them to also setup one extra program is more than reasonable imo. I don't think we have to baby (for lack of a better word) new contributors if they're trying to contribute to stuff like this

[ashquarky]

Okay, so I think we have a provisional plan here:

• Centralise the URLs in Inkay to allow changing the base domain (pretendo.cc) easily
• Investigate using Cemu for Juxtaposition development ( [Feature]: Port to PC and/or Cemu graphics pack Martini#13 )
• Clean up Meowth and add support for SSL patching the applets
• Document an ideal workflow on the developer wiki using the above points
• Other workflow / automation stuff where we can (e.g. pretendo-docker automatically hosting mitmproxy for you)

[jonbarrow]

That sounds good to me. I think long-term maybe we should look into having a monorepo people can pull down that has things like:

• mitmproxy-nintendo (Charles my beloved, but it is paid software)
• A custom Inkay builder (maybe with a CLI that just takes in some parameters to build Inkay with? Like the domain and such)
• The server repos themselves (git submodules?) to be worked on
• Some way to spin up the network easily from the local servers (can Docker do this cleanly?)

I've never designed anything like this before so I'm a tad out of my element here, but roughly something like:

developer-environment/
├── patches/
│   ├── wiiu/
│   │   ├── builder/
│   │   └── inkay/
│   └── 3ds/
│       ├── builder/
│       └── nimbus/
├── proxy/
│   └── mitmproxy-nintendo files here
├── repositories/
│   ├── account/
│   ├── nex-go/
│   ├── miiverse-api/
│   └── etc
└── orchestration/
    └── server runner idk

[ashquarky]

This sounds good longer-term. I might suggest that running the server binary itself on the host should be a supported case in such a setup (rather than doing a docker build cycle) since it allows for improved IDE integrations and using a debugger and this type of thing, which would be harder if the server under development is inside Docker. We're creeping pretty far out of scope for an Inkay issue though.

(That won't be a problem fwiw, it just needs to spit out a .env file with the right keys and docker IP addresses)

[jonbarrow]

I assumed we were already pretty out of scope for this issue already to be fair, and had just started using it as a place to discuss the dev env more broadly/publicly

Also, it should be noted that "server binary" isn't necessarily a thing in all places. Anything using TypeScript/JavaScript won't have a binary to run. I don't disagree with the premise this is just something to keep in mind

Will and jvs may have more opinions on this sort of long-term setup though, this sounds like something more their speed