Beyond a Raspberry Pi 500 …

If you want to try to get better results than I was able to produce, consider using a Raspberry Pi 5 with the maximum RAM configuration (16GB), an add-on board for SSD storage and a case with integrated fan. 

As the market for memory has been wreckedsignificantly impacted by the AI industry, this will push the project sadly beyond the line where I would consider it 'low-cost', landing it somewhere up to €/$400.

It might be a better idea to look out for cheap used mini PCs or just to work with hardware you already have. 

The more memory you have, the better the results you'll be able to get. Larger local language models reduce errors, hallucinations and improve recognition of accents and specialist language. With sufficient hardware, speaker diarization — distinguishing between different speakers — also becomes feasible.