Automated Prompt Optimisation with DSPy
Automated Prompt Optimisation with DSPy
and an update on Trelis Discord Access
Trelis Discord - Lifetime Access
I’m moving Discord access from a monthly subscription to lifetime access. Membership to the Trelis Discord is now available for either:
A one-off purchase, or
As part of the Trelis Multi-Repo (ADVANCED-fine-tuning, ADVANCED-inference, ADVANCED-vision, ADVANCED-speech) + Discord Bundle.
Automating Prompt Engineering with DSPy
There are lots of "prompt engineering tricks"... including few-shot examples... requesting the model to "think step by step".
These work well, but it takes trial and error to work out a great prompt.
DSPy makes prompt generation programatic. So it saves you time as a dev, and it generates prompts that are more robust across a wider range of tasks.
I used DSPy to speed-run up through the classic prompt engineering rules of thumb - and it significantly improved accuracy.
🔍 DSPy speedrun:
Basic Q&A: 20% accuracy
Chain of Thought: 34% accuracy
Retrieval Augmented Generation (RAG): 42% accuracy
Few-shot examples: 54% accuracy
Optimized few-shot examples: 58% accuracy
Multi-hop retrieval with optimized examples: 68% accuracy
💡 Advanced Features:
- Bootstrap Few-Shot: Optimizes example selection based on training data performance
- Multi-hop Retrieval: Improves answer quality through iterative searching
- Assertions: Ensures responses meet specific criteria (e.g., length, format)
And here's the full vid:
with thanks to Tom Dörr for tips on DSPy usage!
And that’s it for this week.
Cheers, Ronan
Trelis.com