The Fifth Elephant 2025 Annual Conference (18 & 19 July)

In the field of software development, the incorporation of AI into coding is transforming the way developers engage with their codebases. Atlassian’s Rovo Dev CLI is leading this charge, providing a cutting-edge AI-powered command line tool that boosts productivity by automatically generating code. It seamlessly integrates with the Atlassian software development ecosystem, delivering enhanced context awareness compared to traditional models.

What is Rovo Dev CLI

• A command line tool that generates code using advanced AI models, code intelligence, and deep integration with local codebases and Atlassian tools.
• Developers use natural language prompts in the terminal (e.g., “Generate a Python function for X” or “Add unit tests for this file”).
• Supports both interactive and non-interactive modes for a conversational experience in generating and refactoring code, and integrating with Atlassian tools like Jira, Confluence, and Bitbucket.

Anatomy of Rovo Dev CLI

• Understanding Codebase: Integrates with repositories, using local memory to understand codebase structure, dependencies, and coding patterns.
• Context Awareness: Combines information from code and related Jira issues to generate contextually appropriate code. Integrates with MCP servers for better context awareness (e.g., “what tasks are assigned to me?”, “pick task 3 and generate code plan”, “mark task as completed”).
• Plan Generation: Outlines necessary files and changes before generating actual code.
• Code Generation & PR: Facilitates code generation and pull requests.

Quality

• SWE Benchmark: An offline evaluation framework developed using publicly available GitHub repositories.
• Statsig as a comprehensive multi-layer experimentation framework, enabling the execution of numerous experiments simultaneously while minimizing costs.
• Impact: 41.98% resolve rate across 2294 tasks.

Enhancing Coding Agents

• Data: Off-the-shelf LLMs are typically trained using publicly available datasets, such as those measured in SWE bench. To enhance the performance of LLMs, it is essential to focus on curating valuable datasets that are not accessible in the public domain.
• Tools: Focuses on enhancing the agent-computer interface and acquiring essential information that enables agents to execute their tasks effectively, such as retrieving context from Jira and generating documents in Confluence.
• Context: This is primarily a search-related challenge stemming from the limitations of the context window in large language models (LLMs). Evaluation of solutions involve offloading search context to external systems to enhance performance.
• Behavior: This entails crafting precise prompts that help clarify requirements, facilitate validation, and enable self-correction.

Impact

• New code generation is 80% faster.
• Refactoring to a new domain model is approximately 50% faster.

Future Direction

Evolution from Co-Pilot to Auto-Pilot to Pilot.

About the presenter

Bala Nathan, Senior Principal Engineer, Atlassian