Installing and Building the Right whisper.cpp

Cloning the Github repository locally

cd ~
git clone https://github.com/ggerganov/whisper.cpp

Build additional libraries

sudo apt install -y libsdl2-dev

Building whisper-stream

Here, you'll have to check what and how much is built to keep the Raspberry Pi from crashing.
This worked on my Raspberry Pi 500:

cd ~/whisper.cpp
rm -rf build
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DWHISPER_SDL2=ON
cmake --build build --target whisper-stream -- -j2