sdm with pishrink and dd

[mcguirepr89]

I think I would like to start testing sdm for use when creating pre-installed images for BirdNET-Pi.

My process right now is (crap):

1. Install RaspiOS and set password, locale, and perform updates, (basic piwiz stuff) (sdm can do for me)
2. Install and configure autoAP (sdm can replace some of this?)
3. Run my BirdNET-Pi installer against a default config file (sdm can do for me probably?)
4. Make the desktop look pretty (sdm can do this for me)
5. Get pishrink.sh (sdm can definitely do for me)
6. Manually remove my info from /etc/wpa_supplicant/wpa_supplicant* (sdm can definitely do for me)

Then I mount it all on another linux machine to truncate (via gparted) the unused space before creating an image.

I've been using a customized pishrink.sh for shrinking the image afterwards.

I suspect I would be able to tuck the dd image creation and pishrink.sh into an sdm-customphase? Is that the idea? Does that seem plausible?
(I'm sure the sdm-customphase would also need to handle mounting the external storage where the image will be stored as well.)

[gitbls]

Great questions. Here are my thoughts on the points of your process:

1. Yes, sdm can do all of that quite handily
2. You can put code in your sdm-customphase Phase 0 to copy the autoap files from a local location, or even curl/download them from github. Either way is fine. Phase0 has complete access to the whole world, as well as your IMG. Just make sure to prefix accesses you want to be in the IMG with "/mnt/sdm" so you don't modify your running system!
3. I'm not familiar with the BirdNet-Pi installer. would have to take a look at it to see if you can use it as is, or whether some work might be required to a) copy files into the IMG during sdm Phase 0 and then install them in Phase 1. Do you have a pointer to the installer?
4. sdm can definitely do that. You didn't say which desktop you're using, but you have full access, and sdm has a few switches to help you stuff pre-built LXDE config files into the IMG if you're using LXDE. I primarily use icewm, and I simply copy my config files from local storage into the IMG during Phase 0.
5. I believe that pishrink will need to be run Outside of sdm, since it operates on the IMG file itself. I made a note to look at pishrink in more detail, but will probably be in early December due to the Thanksgiving holiday here in the US.
6. Why manually remove it? Just don't put your wpa_supplicant in the IMG at all! You can have sdm stuff it onto the SD Card when you burn it.

What "dd image creation" are you referring to? That's not clear to me. See #5 above re pishrink.

sdm-customphase isn't the right place to mount external storage containing an IMG, since sdm checks that the IMG is accessible long before sdm-customphase runs. Best practice would be to wrap sdm inside a script that mounts the external storage, calls sdm as desired, and then dismounts the storage.

I have my IMGs on an NFS-served SSD that all my systems automount. The performance is quite good, although if I need to start with a fresh "virgin" IMG I run a remote ssh on the NFS server to avoid sending the entire IMG across the network two extra times. I'm quite impatient like that 🙄

HTH. If something isn't clear, ask away and I'll be happy to answer.

[mcguirepr89]

Thank you so much for your detailed and helpful response.

Do you have a pointer to the installer?

This is the core installer. I'm embarrassed at my spaghetti code compared to your scripting, BUT I'm sure you'll be able to follow along.
The script accepts a preconfigured birdnet.conf file, so I just manually install the system like this:

git clone https://github.com/mcguirepr89/BirdNET-Pi.git /home/pi/BirdNET-Pi
cp /home/pi/BirdNET-Pi/birdnet.conf-defaults /home/pi/BirdNET-Pi/birdnet.conf && /home/pi/BirdNET-Pi/scripts/install_birdnet.sh <<EOF

EOF

The installer is interactive even when fed a config file as it asks the user to confirm (press Enter) that they have read and agree to the LICENSE, so that is the only reason for the heredoc.

The installation is basically

1. apt getting dependencies (though I see that you always use apt-get, so I suppose I ought to probably understand why and shift things over to that),
2. a hand full of systemd services are created and enabled (not started),
3. and a python virtual environment is created to install the python site packages needed for BirdNET-Lite (the TFLite model on which my project is built).

I believe that pishrink will need to be run Outside of sdm, since it operates on the IMG file itself. I made a note to look at pishrink in more detail, but will probably be in early December due to the Thanksgiving holiday here in the US.
Why manually remove it? Just don't put your wpa_supplicant in the IMG at all! You can have sdm stuff it onto the SD Card when you burn it.

Things make better sense to me now and certainly pishrink.sh should be a separate process.

What "dd image creation" are you referring to? That's not clear to me. See #5 above re pishrink.

I forgot to mention that step, which is an integral part of my process of creating the image. Since I do all of the customization from within the OS itself, I end up creating an image from those customizations using sudo dd if=/dev/mmcblk0 of=/media/pi/somemounteddisk/birdnet-pi_version.img bs=1M count=8000 status=progress after shrinking the rootfs partition to below 8GiB.

Using sdm, however, I would be altering the image directly, so I won't need to dd anything to create a new image. Sorry to be unclear there.

Let me know what you think about the BirdNET-Pi installer!

My best regards,
Patrick

[gitbls]

I took a quick look at the scripts (not bad, btw!).

My suggestion is to just dive into it: do an sdm --explore into a test copy of your IMG, copy all the files you need into it (using the network...git or scp, since you can't access the local system), run the scripts and see what happens. It may just work, or there may be bits of breakage. Rummage through the breakage and sort out the issues (happy to help if you get stuck).

(sdm --mount is like sdm Phase0, and sdm --explore is like sdm Phase1, from the perspective of how the environment is configured).

Aside: I'm thinking of adding a --shrink switch to sdm that will do a basic "shrink the image". I think it would be a useful addition, and allow you to make the --extend size plenty huge but with zero ultimate burn-time cost. It won't have all the pishrink bells and whistles, but will enable you to use sdm for the basic shrink capability. I have a few other things to tend to as well, so not sure when I'll get to it, but from your perspective it will just be an operational change.

[mcguirepr89]

Thanks for taking a look -- I will go for it sometime this coming week and will let you know how I faire.

Using sdm certainly seems like something worth pursuing from my vantage. And, although it is great what pishrink.sh can do, most of what I use it for is the compression (just xz) and the customizations I've injected into the script (cleanup stuff). I'm not a huge fan of how it handles the expansion after first boot and have been considering writing my own version of pishrink.sh that just uses the clever loop mounting, and the rest of it would be replaced. I suspect the simplified --shrink option you're working on would be better suited to my needs, though.

Thanks for your time today and for your contributions to my project. If you find any more interest in BirdNET-Pi, I currently have a pretty active testing community and we're really focused on getting the user used to autoAP using systemd-networkd over dhcpcd. I've fudged things here and there for the configuration wizard to temporarily restart dhcpcd so that users can configure the wpa_supplicant-wlan0.conf from the Wireless & Wired Network panel applet. One of the hurdles is simply in getting the average bird enthusiast to understand Access Points well enough to use a VNC client for internetless installations/configurations and headless installations.
Drop by the "Testing" discussion anytime and take a look at how my users are responding to autoAP -- you'll likely see some more comments regarding autoAP later today as there are other testing responses pending from my most recent image release yesterday.

My best,
Patrick