How to start with UI on the Mac?

[d5364t54ytfr4]

Like are there any libraries/headers for working with UI?

[autc04]

You will want to browse old documentation about the "Window Manager", "Menu Manager", "Dialog Manager" and "Control Manager", and maybe about the "TextEdit package".

For Retro68-specific sample code, look at Samples/Dialog, Samples/WDEF (ignoring the stuff that is actually about WDEFs) and maybe LaunchAPPL/Server; and, especially if you're using Nix, at the https://github.com/autc04/Retro68NixSample repository.

By modern standards, Apple's old UI libraries are very basic and low-level.
There were several object oriented frameworks (Apple's MacApp, Symantec's THINK Class Library, and Metrowerk's PowerPlant) built on top of it, but none of these have been ported to run on Retro68 yet, and doing so is probably not suitable as a first introduction to classic Mac GUI development ;-)