A Holistic Approach to Domain-Specific Small Language Models: Instruction Tuning, Retrieval-Augmented Generation from Multiple Documents
List of Authors
Ashikin Abdullah, Syahid Anuar
Keyword
small language models, domain-specific SLMs, fine-tuning, retrieval-augmented generation
Abstract
While Large Language Models (LLMs) such as GPT and LLaMA demonstrate broad versatility across general tasks, they exhibit significant performance limitations when applied to specialized domains due to inadequate domain-specific pre-training. Industries such as healthcare, finance, legal, and manufacturing require AI systems capable of understanding nuanced terminology, complex regulations, and domain-specific reasoning patterns that generic LLMs cannot reliably provide. Furthermore, training domain-specific models from scratch incurs prohibitive computational and financial costs, making this approach inaccessible for most organizations. Existing solutions remain fragmented, with domain-specific implementations requiring separate infrastructure for each application area and often relying on external APIs that compromise data privacy and regulatory compliance. This paper proposed a fine-tuned SLM with RAG for domain-specific Small Language Models (SLMs) through instruction-tuning to maximize data privacy, reduce external API dependency, and enable domain-specific customization. The proposed system will address critical domain adaptation challenges, including complex reasoning, retrieval reliability, and bias mitigation. By adopting a fully open and local technology stack, the system ensures transparency, reproducibility, and data sovereignty—crucial for healthcare, finance, and government applications requiring privacy compliance. Current validation efforts focus on evaluating performance and effectiveness across multiple industry applications, with future work providing detailed benchmarks.