AI Strategy for Small Teams: Is Custom Tuning the Answer, or Should We Leverage the Power of General-Purpose Models?

As enthusiasm for AI research and adoption intensifies, resource-constrained small teams often face a dilemma. With limited budgets and personnel, should they dive into developing custom models, or should they strategically leverage rapidly advancing general-purpose models? Recent data shows the performance of general-purpose models is rising sharply, suggesting it's time to reassess the cost-effectiveness of approaches that focus on enhancing overall system logic.

Current Status: Investigated Facts and Data

The performance improvement of major open-source LLMs over the past year is remarkable. Using the Llama series as a benchmark, knowledge capabilities on the MMLU (Massive Multitask Language Understanding) benchmark have improved from about 19% to 51%. In GSM8K, which evaluates mathematical reasoning, performance improvements of up to 200% or more have been recorded. Specifically, the Llama 3 70B model saw its MMLU score rise from 68.9% to 82.0% compared to its predecessor, and even the smaller 8B model showed a dramatic leap from 45.3% to 68.4%.

Research also exists comparing the pace of development of these general-purpose models with the utility of custom approaches. For domain-specific tasks or fields requiring strict output formats, such as healthcare and law, LoRA fine-tuning provides high accuracy and stability. However, for general reasoning tasks with insufficient data, the prevailing analysis is that prompt engineering can be a more cost-effective choice and, in some cases, can achieve similar performance.

Analysis: Meaning and Impact

This data conveys important implications for small teams. Precisely because the development speed of general-purpose models is so fast, concentrating resources on fine-tuning for a specific task may not be economical in the long term. There is a risk that the performance of a model fine-tuned with today's investment could fall behind the base version of a new general-purpose model released six months later.

Therefore, the core strategy should focus on 'enhancing overall system logic' and 'strategic selection of LLM integration points'. Instead of having the LLM solve every problem, it's crucial to maintain existing solid business logic and processes while precisely integrating the LLM at points where it can shine brightest (e.g., natural language understanding, creative idea generation, complex knowledge queries). This is an approach that focuses on designing a higher-level system that effectively utilizes the model, rather than modifying the model itself.

Practical Application: Methods Readers Can Utilize

Small teams should first clearly separate their project requirements. Tasks where output formalization and domain expertise are key (drafting contracts, summarizing specific medical papers) can be strong candidates for LoRA fine-tuning. On the other hand, tasks like free-form conversation, general analysis, and idea brainstorming are highly likely to be sufficiently solvable with general-purpose models through prompt engineering and advanced logic (chaining, agent patterns).

Furthermore, it's important to recognize the lack of clear standards for ROI measurement and to establish the team's own practical evaluation criteria. After measuring the baseline of existing workflows, comparing metrics like time required, accuracy, and user satisfaction through AI adoption pilots in an A/B test format can be a starting point. Setting metrics directly linked to business goals (reducing customer response time, shortening content creation cycles) is more meaningful than simply measuring performance improvement per investment.

FAQ

Q: Given that the performance of general-purpose models is rising so quickly, is there any point in small teams learning fine-tuning? A: It still makes sense for tasks in specific domains (healthcare, law, finance) where strict accuracy and format compliance are critical. However, for general tasks, it may be more practical to prioritize developing skills in effective prompt design and building AI agent systems over acquiring fine-tuning techniques.

Q: How can we scientifically decide whether to introduce an LLM into a project or not? A: There is no single standardized methodology agreed upon worldwide. The most practical method is to clearly define key performance indicators (KPIs) before and after introduction and measure their impact quantitatively through small-scale pilots. Comparing the value gained versus cost (time saved, errors reduced, revenue generated) against existing methods forms the basis.

Q: Are there general criteria for when prompt engineering alone can replace LoRA tuning? A: No clear criteria covering all tasks have been established by research. However, if it's difficult to secure sufficient high-quality training data for the task, or if the task definition is flexible and requires creativity, it's highly likely that prompt engineering and enhanced system logic can yield sufficient results.

Conclusion

For small teams to succeed in AI research and adoption, they must shift from a technology-centric approach chasing the latest models to strategic thinking. It is wise to acknowledge the steep growth curve of general-purpose models and focus precious resources on designing higher-level systems that effectively control and utilize them, and on points directly linked to creating business value, rather than on fine-tuning the models themselves. Ultimately, the key is not in making the most powerful tool, but in knowing how to use that tool to do the most valuable work.

참고 자료

• 🛡️ Comparison of Prompt Engineering and Fine-Tuning Strategies in Large Language Models in the Classification of Clinical Notes
• 🛡️ Challenges in calculating the ROI to Generative AI for financial institutions - Columbia SIPA
• 🏛️ The Llama 3 Herd of Models
• 🏛️ Llama 2: Open Foundation and Fine-Tuned Chat Models
• 🏛️ Toward an evaluation science for generative AI systems