In this guide
01Proprietary vs Open-Source
02RAG vs Fine-tuning
03Single Agent vs Multi-Agent
04Sync vs Async Inference
05Vector vs Full-text vs Hybrid
Need help making these decisions for your product?
Talk to an engineer →
Decision 01
Proprietary LLM vs Open-Source Model
Use Proprietary (GPT-4o, Claude, Gemini)
When performance, speed, or multimodality is the priority
- You need state-of-the-art reasoning without managing GPU infrastructure
- Your use case involves complex tasks, long context, or image/audio inputs
- You're prototyping fast and don't have ML infrastructure expertise in-house
- Token cost at your current volume is acceptable relative to build overhead
Use Open-Source (Llama, Mistral, Falcon)
When data residency, control, or cost at scale is the constraint
- Data privacy or regulation prevents sending data to third-party APIs
- You need full control over fine-tuning, model behaviour, or output format
- Per-token pricing at your projected volume makes SaaS APIs unsustainable
- You're building a product where the model itself is part of the IP
Key question
"Can your use case tolerate a third-party API dependency, or do data residency requirements or cost at scale require self-hosting?"
Decision 02
RAG vs Fine-tuning
Use RAG (Retrieval-Augmented Generation)
When the challenge is what the model knows
- Your knowledge base changes frequently — docs, policies, product data
- You need citations or source attribution alongside the answer
- You want to add domain knowledge without ML infrastructure or labelled data
- You need the model to answer questions about specific, private documents
Use Fine-tuning
When the challenge is how the model responds
- Style, tone, or output format must be tightly and reliably controlled
- Your task is narrow and well-defined with abundant labelled examples
- Latency or cost of multi-step RAG retrieval pipelines is a constraint
- You're teaching a new task, not adding new knowledge
Key question
"Is your challenge about what the model knows, or how it responds? RAG changes the input; fine-tuning changes the model."
Decision 03
Single Agent vs Multi-Agent System
Use a Single Agent
When the task is bounded and latency matters
- Your task has a clear, predictable sequence of steps with a single goal
- You want simplicity: one prompt, one decision path, one output
- Latency is critical — coordination overhead between agents is unacceptable
- You're building a v1 and want to validate the core loop before scaling complexity
Use Multi-Agent
When tasks are parallelisable or need specialist checking
- Subtasks can run concurrently and reduce wall-clock time
- Different parts of the task require different specialised capabilities or tools
- Adversarial checking adds value — one agent validates another's output
- Different parts of the workflow have different latency or reliability requirements
Key question
"Would a single senior analyst handle this end-to-end, or would they hand it off to specialists? Match the system topology to the human workflow."
Decision 04
Synchronous vs Asynchronous Inference
Use Synchronous (blocking)
When the user must wait for the result immediately
- Users expect a real-time response in a chat or autocomplete interface
- The output feeds directly into the user's next action or decision
- Your latency SLA is under 5–10 seconds and the model can reliably meet it
- The inference call is short enough that queuing adds more complexity than value
Use Asynchronous (queued)
When the output can be delivered later
- Tasks take longer than a user reasonably waits — document analysis, batch runs
- The output is consumed later — reports, summaries, overnight processing
- You want to decouple inference load from your web tier for reliability and cost
- Retry logic and failure handling are easier at the queue level than in-request
Key question
"Does the user need to wait for this result to continue their task, or can it arrive in the background? If they can continue without it, go async."
Decision 05
Vector Search vs Full-Text Search vs Hybrid
Vector DB (Pinecone, pgvector, Weaviate)
Semantic similarity first
- Queries are natural-language questions, not structured lookups
- Semantic meaning matters more than keyword match
- You're building a RAG pipeline and need nearest-neighbour retrieval
- Content is unstructured — documents, notes, knowledge bases
Full-Text Search (Elasticsearch, Typesense)
Precision and recall on exact terms
- Users expect keyword-level precision — names, codes, product IDs
- You need faceted filtering and aggregations alongside search
- Your corpus is updated in near real-time and needs low indexing latency
- The search is structured — e-commerce, support tickets, logs
Hybrid (vector + keyword)
Maximum recall across both modes
- Users ask both semantic questions AND keyword-level queries
- Recall must be maximised — missing a relevant result is costly
- You have the infrastructure maturity to manage two retrieval systems
- You're serving a diverse user base with different search mental models
Key question
"Are users asking meaning-driven questions, looking up specific items by name or ID, or both? The answer tells you which retrieval mode to prioritise."
Working through these for your product?
TechTek can validate your architecture before you build
We've made these decisions across dozens of AI products. In a 30-minute call we can tell you which path fits your constraints — and flag anything that tends to cause pain six months later.
Book a free architecture review →