Post

LLAMOSC: GSoC '24 OREL INCF Project Report

On the design and implementation of LLAMOSC, a framework that pairs LLM-driven agents with agent-based modeling to simulate open-source community sustainability.

A Bird’s-Eye Overview

This project focused on Open Source Community Sustainability, building a framework that uses an agent-based model powered by Large Language Models (LLMs) to simulate community interactions and encourage sustainable practices. The work extends earlier efforts from GSoC ‘22 and ‘23, integrating system- and agent-based modeling with LLMs to simulate open-source communities with greater fidelity.

Need for the Project

As demand for open-source software grows, so do the challenges of community management and collaboration. LLM-based agents possess advanced reasoning, decision-making, and adaptive learning capabilities, which allows them to function as intelligent entities within simulations. The framework aims to map the complexity of community engagement and offer insights into effective management and sustainability strategies.

The Final Framework: LLAMOSC

Rather than a single-use script, the project produced a reusable framework intended for broader application across many open-source projects.

LLAMOSC — LLM-Powered Agent-Based Model for Open Source Communities — is a framework designed to simulate and sustain open-source communities using advanced LLM and Agent-Based Modeling (ABM) techniques:

  • LLM — provides intelligent agents with perception, reasoning, and decision-making capabilities
  • ABM — uses agent-based modeling to simulate community interactions and dynamics
  • OSC — focuses on open-source communities to enhance sustainability, collaboration, and engagement

How LLAMOSC Works

The framework operates through a simulation structure that includes a GitHub CodeSpace environment, environment variables, multiple agents, and agent-specific internal states.

Implemented Components and Features

1. Environment

GitHub (CodeSpace Environment). A simulated code space powered by AutoCodeRover where agents contribute to projects, mirroring real GitHub repositories.

2. Environment Variables

  • Issues — categorized by difficulty levels 1–5, ranging from beginner-friendly “Good First Issues” to “Expert Required Issues,” assigned based on contributor experience.
  • Code Quality — tracks the average code quality of pull requests, measured and updated at each simulation step with trend visualization.

3. Agents

  • Coding Ability — skill level in writing and reviewing code, powered by AutoCodeRover, affecting solution effectiveness and code-quality contributions.
  • Role Description — each agent receives a unique backstory, influencing initial motivation and behavior during GitHub discussions.

4. Agent Variables (Internal States)

  • Experience Level — categorized 1–5 (Novice → Experienced), tracked and plotted over time to show contributor growth.
  • Motivation Level — shaped by role descriptions and influenced by task acceptance/rejection, issue outcomes, and code-quality reviews.

Framework Capabilities

AutoCodeRover Integration

Combines LLMs with code navigation and debugging through two stages: context retrieval and patch generation. Features Program Structure Aware code-search APIs for codebase navigation.

Automated Pull Request Lifecycle

  • ContributorAgent automates issue identification, solution proposal, and pull-request creation.
  • MaintainerAgent automates pull-request review and merging based on predefined criteria.
  • Docker is used to keep environments consistent and isolated.
  • Diff extraction and automated PR submission are fully scripted.

Multi-Agent Decision-Making Algorithms

  • Authoritarian (Benevolent Dictator). A central maintainer manages and allocates tasks; contributors are rated and assigned at the maintainer’s discretion.
  • Decentralized (Meritocratic). Distributed decision-making among contributors, who bid on tasks based on suitability and experience.

Metrics are visualized throughout the simulation at each time step — including contributor experience levels, code-quality trends, and motivation levels.

Accomplishments So Far

  • Preliminary design and implementation of a framework for simulating open-source environments with agents reflecting real-world skill variation.
  • Automated pull request lifecycle — streamlined issue identification, solving, PR creation, submission, review, and merging.
  • Multi-agent decision making — algorithms for both centralized and decentralized governance models.
  • Graphical user interface — an intuitive interface for input and metric visualization.

Future Improvements

  1. Collaboration algorithm for multiple agents — enable multiple agents to work together on a single issue to improve efficiency and reflect real-world teamwork.
  2. Dynamic issue creation — an IssueCreatorAgent that introduces new issues throughout the simulation rather than only at startup.
  3. ConversationSpace — simulate Slack-like communication for more accurate team collaboration.
  4. RAG integration — add Retrieval-Augmented Generation within conversation spaces and GitHub discussions for improved context-aware interactions.
  5. Engagement metrics — measurement systems for interaction quality and participant involvement within conversation spaces.

Acknowledgements

Thanks to the Orthogonal Research Lab community for the collaborative environment and inclusivity throughout the project — weekly meetings and discussions with experienced researchers and fellow contributors made this experience particularly rewarding. I look forward to continuing contributions to both the project and the lab.

References

Originally published on OREL Medium, August 24, 2024.