Salesforce AI Unveils SWERank: A Cost-Effective Solution for Precise Software Issue Localization

Challenges in Software Issue Localization

Identifying the exact location of a software issue, such as bugs or feature requests, remains a highly time-consuming part of the development process. While automated patch generation and code assistants have made progress, finding the precise code area that requires a change often takes more effort than deciding how to fix it. Agent-based methods using large language models (LLMs) simulate developer workflows but tend to be slow, fragile, and costly, especially when relying on proprietary models. Existing code retrieval models are faster but not optimized for the detailed and behavior-focused nature of real-world issue descriptions, creating a gap between natural language inputs and effective code search.

Introducing SWERank: A Practical Framework

Salesforce AI has introduced SWERank, a lightweight and efficient retrieve-and-rerank framework designed specifically for software issue localization. SWERank reframes the localization task as a code ranking problem with two main components:

• SWERankEmbed: A bi-encoder retrieval model that encodes GitHub issues and code snippets into a shared embedding space for similarity-based retrieval.
• SWERankLLM: A listwise reranker built on instruction-tuned LLMs that refines the ranking of retrieved code candidates by leveraging contextual understanding.

The system is trained on SWELOC, a large-scale dataset extracted from public GitHub repositories linking real-world issue reports with corresponding code changes. SWELOC uses contrastive training with consistency filtering and hard-negative mining to ensure high-quality, relevant training data.

Architecture and Methodology

SWERank operates in two stages. First, SWERankEmbed encodes issue descriptions and candidate functions into dense vectors. It uses a contrastive InfoNCE loss to increase similarity between an issue and its related function while reducing similarity to unrelated snippets. Hard negatives—semantically similar but irrelevant code—are carefully mined to improve the model’s discrimination.

Next, SWERankLLM reranks the top-k code candidates by jointly processing the issue description and candidates to produce a ranked list prioritizing relevant code. The reranker is trained to identify the correct code snippet using a simplified supervision approach compatible with listwise inference.

This design enables high performance without multiple interaction rounds or expensive agent orchestration.

Performance and Efficiency

Evaluations on benchmark datasets SWE-Bench-Lite and LocBench demonstrate that SWERank achieves state-of-the-art accuracy across file, module, and function levels. For example, SWERankEmbed-Large (7B parameters) reached an 82.12% accuracy@10 at the function level, outperforming competitors like LocAgent with Claude-3.5. When combined with SWERankLLM-Large (32B parameters), accuracy rose to 88.69%, setting a new standard.

SWERank also offers significant cost savings. While Claude-based agents cost approximately $0.66 per example, SWERankLLM inference costs $0.011 for the 7B model and $0.015 for the 32B model, offering up to 6 times better accuracy-to-cost ratio. Moreover, the lightweight SWERankEmbed-Small model with 137M parameters achieves competitive results, showcasing scalability and efficiency.

Broader Impact and Future Potential

Fine-tuning various embedding and reranking models on the SWELOC dataset yields considerable accuracy improvements, confirming SWELOC’s value as a training resource for issue localization. By modeling localization as a ranking problem rather than relying on complex agent interactions, SWERank presents a practical, scalable solution for debugging and code maintenance.

Salesforce AI’s approach demonstrates that open-source tools can deliver high accuracy, efficiency, and ease of deployment in automated software engineering, setting a new benchmark for the field.

Additional Resources

For those interested, the full research paper and project details are available online. Follow the researchers on Twitter and join communities like the ML SubReddit to stay updated on the latest in AI and machine learning advancements.