CloodGroups: Revolutionizing Documentation Through Automated Git Commit Analysis

TL;DR

We developed CloodGroups, an innovative system that automatically generates documentation from git commits by reverse-engineering the prompts that would recreate the code changes. This approach preserves development context, creates living documentation, and significantly improves knowledge sharing. This article details how we implemented the system, the database schema that powers it, our structured approach to documentation generation, and real-world examples of CloodGroups in action.

The Documentation Challenge

Software documentation has always been a challenge. Traditional approaches suffer from several problems:

• Documentation Drift: Documents quickly become outdated as code evolves
• Context Loss: The reasoning behind implementation decisions gets lost
• Tedious Maintenance: Keeping documentation in sync with code requires significant effort
• Knowledge Silos: Critical insights remain trapped in specific team members’ heads
• AI Context Limitations: AI tools like Claude lack the “why” behind code changes

These issues lead to poor knowledge transfer, reduced development velocity, and increased technical debt. We needed a solution that would make documentation an organic part of the development process rather than a separate, often-neglected task.

Enter CloodGroups: A New Approach

We created CloodGroups as a revolutionary approach to documentation that automatically generates structured documentation from actual code changes. CloodGroups preserve not just what changed, but the reasoning and intent behind those changes.

What is a CloodGroup?

A CloodGroup is a structured JSON file containing:

1. A list of modified files in a meaningful set of changes
2. A descriptive prompt that would theoretically recreate those changes

Here’s a simple example:

{
  "files": [
    "Neuro.Client/components/Tasks/TaskProgress.vue",
    "Neuro.Client/composables/useTaskUpdates.ts"
  ],
  "prompt": "Create a TaskProgress component that displays a progress bar for long-running tasks. It should subscribe to SSE notifications using the useTaskUpdates composable to receive real-time progress updates. The component should show percentage complete, estimated time remaining, and current status."
}

This approach captures not just the files that changed, but the intent and purpose behind those changes. It’s like preserving the requirements that led to the implementation.

The CloodGroups Technical Implementation

System Architecture Overview

Our CloodGroups system consists of several interlinked components:

┌─────────────────┐      ┌─────────────────┐      ┌─────────────────┐
│                 │      │                 │      │                 │
│   SQLite DB     │◄────▶│  Analysis Engine│─────▶│  JSON Generation│
│   (Task System) │      │  (Git Analysis) │      │                 │
│                 │      │                 │      │                 │
└────────┬────────┘      └─────────────────┘      └────────┬────────┘
         │                                                  │
         │                                                  │
         ▼                                                  ▼
┌─────────────────┐                                ┌─────────────────┐
│                 │                                │                 │
│ Feature Tracking│                                │ CloodGroup Files│
│                 │                                │                 │
└─────────────────┘                                └─────────────────┘

1. SQLite Database Schema

The foundation of our CloodGroups system is a well-structured SQLite database that tracks tasks, features, and their related documentation:

-- Clood Groups table
CREATE TABLE clood_groups (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    item_id INTEGER NOT NULL,
    feature_id INTEGER,
    file_path TEXT NOT NULL,
    prompt_content TEXT NOT NULL,
    files_included TEXT,  -- JSON array of file paths
    created_date DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
    status TEXT NOT NULL DEFAULT 'generated',
    start_commit TEXT,
    end_commit TEXT,
    FOREIGN KEY (item_id) REFERENCES items (id),
    FOREIGN KEY (feature_id) REFERENCES features (id)
);

-- Settings for automatic clood group generation
CREATE TABLE auto_clood_group_settings (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    enabled BOOLEAN NOT NULL DEFAULT 1,
    output_directory TEXT NOT NULL DEFAULT '/root/test-claude2/wharfer/clood-groups/',
    template TEXT NOT NULL,
    diff_command TEXT NOT NULL DEFAULT 'git diff --name-status {start_commit} {end_commit}'
);

-- Default settings for clood group generation
INSERT INTO auto_clood_group_settings (template) 
VALUES (
'{
  "files": $FILES,
  "prompt": "Implement a $FEATURE_NAME feature that $FEATURE_DESCRIPTION. This implementation should include $REQUIREMENTS and follow project standards. The solution should be well-tested and documented."
}'
);

-- Procedure table for clood group generation process
CREATE TABLE clood_group_generation_procedures (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    name TEXT NOT NULL,
    description TEXT NOT NULL,
    steps TEXT NOT NULL,  -- JSON array of procedure steps
    enabled BOOLEAN NOT NULL DEFAULT 1,
    created_date DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
);

-- Default procedure for reverse-engineering prompts
INSERT INTO clood_group_generation_procedures (name, description, steps)
VALUES (
    'reverse_engineer_prompt', 
    'Generate a clood group prompt by analyzing git repository changes',
    '[
        {"step": 1, "action": "get_modified_files", "command": "git diff --name-status {start_commit} {end_commit}"},
        {"step": 2, "action": "analyze_file_changes", "command": "git diff {start_commit} {end_commit} {file_path}"},
        {"step": 3, "action": "extract_key_features", "description": "Extract main features and functionalities added"},
        {"step": 4, "action": "generate_prompt", "description": "Generate a prompt that would recreate the changes"},
        {"step": 5, "action": "create_json_file", "template": {"files": "{files_array}", "prompt": "{generated_prompt}"}}
    ]'
);

This schema allows us to:

• Track CloodGroups for specific tasks and features
• Configure automatic generation settings
• Define procedural steps for analyzing git changes
• Store the generated prompts and file lists

2. Git Analysis Engine

When a task is completed, our system analyzes the git repository to identify changes between the start commit and end commit:

#!/bin/bash

# Extract task information from database
TASK_ID=$1
TASK_INFO=$(sqlite3 /path/to/database.db "SELECT name, start_commit, end_commit FROM items WHERE id = $TASK_ID")
TASK_NAME=$(echo $TASK_INFO | cut -d'|' -f1)
START_COMMIT=$(echo $TASK_INFO | cut -d'|' -f2)
END_COMMIT=$(echo $TASK_INFO | cut -d'|' -f3)

# Get list of modified files
MODIFIED_FILES=$(git diff --name-status $START_COMMIT $END_COMMIT | awk '{print $2}')

# Convert to JSON array format
FILES_JSON="["
for file in $MODIFIED_FILES; do
  FILES_JSON+="\"$file\","
done
FILES_JSON=${FILES_JSON%,}  # Remove trailing comma
FILES_JSON+="]"

# Extract feature information for this task
FEATURES=$(sqlite3 /path/to/database.db "
  SELECT name, description 
  FROM features
  WHERE item_id = $TASK_ID AND status = 'completed'
")

# Generate prompt based on features and changes
PROMPT="Implement $TASK_NAME that"

# Add feature descriptions to prompt
for feature in $FEATURES; do
  FEATURE_NAME=$(echo $feature | cut -d'|' -f1)
  FEATURE_DESC=$(echo $feature | cut -d'|' -f2)
  PROMPT+=" includes $FEATURE_NAME: $FEATURE_DESC,"
done
PROMPT=${PROMPT%,}  # Remove trailing comma

# Generate CloodGroup file
FILENAME=$(echo $TASK_NAME | tr ' ' '-' | tr '[:upper:]' '[:lower:]').json
CLOOD_FILE_PATH="/path/to/clood-groups/$FILENAME"

cat > $CLOOD_FILE_PATH << EOF
{
  "files": $FILES_JSON,
  "prompt": "$PROMPT"
}
EOF

# Update database with CloodGroup information
sqlite3 /path/to/database.db "
  INSERT INTO clood_groups (item_id, file_path, prompt_content, files_included, start_commit, end_commit) 
  VALUES ($TASK_ID, '$CLOOD_FILE_PATH', '$PROMPT', '$FILES_JSON', '$START_COMMIT', '$END_COMMIT')
"

echo "CloodGroup generated at $CLOOD_FILE_PATH"

This script is simplified for clarity, but it demonstrates the core logic of our CloodGroup generation process. In practice, we use a more sophisticated system that can:

• Analyze the content changes in each file, not just which files changed
• Identify patterns and common changes across files
• Use AI to generate more descriptive and accurate prompts
• Handle edge cases like file renames and deletions

3. Advanced Prompt Generation

For more sophisticated prompt generation, we leveraged AI to analyze the git diffs and extract the underlying intent. This process follows several steps:

1. Extract Raw Diffs: Get detailed changes using git diff
2. Group Related Changes: Cluster changes by functionality
3. Identify Key Features: Determine the main capabilities added
4. Generate Natural Language: Create a descriptive prompt that captures intent
5. Verify Completeness: Ensure the prompt covers all significant changes

Here's a more advanced prompt generation algorithm:

def generate_clood_prompt(start_commit, end_commit, task_name, feature_info):
    # Get detailed diff information
    diff_output = subprocess.check_output(
        ['git', 'diff', start_commit, end_commit],
        encoding='utf-8'
    )
    
    # Parse the diff to understand file changes
    file_changes = parse_git_diff(diff_output)
    
    # Group changes by functionality
    change_groups = cluster_changes_by_function(file_changes)
    
    # Extract key features from changes
    key_features = []
    for group in change_groups:
        features = extract_features_from_changes(group)
        key_features.extend(features)
    
    # Combine with known feature information
    all_features = merge_features(key_features, feature_info)
    
    # Generate natural language prompt
    prompt = f"Implement {task_name} that "
    
    # Add feature descriptions
    for idx, feature in enumerate(all_features):
        if idx > 0:
            prompt += " and "
        prompt += f"provides {feature['name']} functionality that {feature['description']}"
    
    # Add implementation details based on identified patterns
    patterns = identify_implementation_patterns(file_changes)
    if patterns:
        prompt += ". The implementation should "
        for idx, pattern in enumerate(patterns):
            if idx > 0:
                prompt += " and "
            prompt += pattern
    
    return prompt

In practice, we often use Claude itself to generate these prompts, creating a virtuous cycle where AI helps document code that will later be used by AI.

Real-World CloodGroup Examples

Let's explore some actual CloodGroups we've generated to see the system in action:

Example 1: VNC Desktop Implementation

{
  "files": [
    "docs/vnc-desktop-task.md",
    "docs/vnc-desktop-design.md"
  ],
  "prompt": "Design a shared Ubuntu Desktop environment accessible via VNC for collaborative development. The system should allow multiple team members to connect simultaneously to the same desktop session, with secure authentication and encrypted connections. Include both a task document outlining requirements and phases, and a technical design document with installation steps, configuration details, and security considerations. The solution should solve the problem of command-line only development by providing a persistent graphical workspace."
}

This CloodGroup captures both the technical requirements and the business purpose of our VNC implementation. It preserves the context that we wanted to solve the limitations of command-line only development through a shared graphical workspace.

Example 2: SSE Connection Optimization

{
  "files": [
    "Neuro.Client/public/sse-worker.js",
    "Neuro.Client/src/services/sseService.ts",
    "Neuro.Client/src/composables/useTaskUpdates.ts",
    "Neuro.Client/src/composables/useJobLogs.ts",
    "Neuro.Client/src/services/sseMonitor.ts",
    "Neuro.Client/run-sse-tests.sh"
  ],
  "prompt": "Optimize Server-Sent Events (SSE) connections by implementing a Service Worker proxy to reduce the number of concurrent connections. Create a Service Worker that establishes one persistent SSE connection and broadcasts messages to interested components. Implement a client-side service layer for managing SSE subscriptions and message routing. Update existing composables (useTaskUpdates, useJobLogs) to use this new service instead of creating their own connections. The solution should reduce the number of SSE connections per browser tab to one, improve scalability, and centralize reconnection logic without requiring changes to the server-side code."
}

This CloodGroup clearly communicates not just the files changed, but the entire architecture of our SSE optimization strategy. The prompt provides a complete roadmap for implementing the feature.

Example 3: Database Setup Improvements

{
  "files": [
    "docs/SETUP.md",
    "Neuro/Migrations/20250302081425_CreateIdentitySchema.Designer.cs",
    "Neuro/Migrations/20250302081425_CreateIdentitySchema.cs",
    "scripts/manage-server.sh"
  ],
  "prompt": "Improve the database setup process for the Wharfer project by: 1) Creating detailed manual setup instructions in SETUP.md with clear steps for database initialization, 2) Adding new database migration files with the correct schema, 3) Creating a server management script (manage-server.sh) that allows starting, stopping, and checking the status of the development server in the background. The documentation should also include HTTPS certificate setup and default login information."
}

This CloodGroup highlights how our system captures infrastructure and setup changes, not just application code. The prompt clearly articulates what the improvements were intended to accomplish.

CloodGroups Timeline and Evolution

Benefits and Impact

Implementing the CloodGroups system has transformed our development process in several significant ways:

Quantitative Benefits

Qualitative Benefits

1. Self-Documenting Codebase: Documentation is generated as a natural byproduct of development
2. Preserved Intent: The "why" behind changes is captured and persisted
3. Improved Knowledge Sharing: New team members can quickly understand the reasoning behind implementations
4. Enhanced AI Context: Claude gets better context for future code changes
5. Consistent Quality: Documentation follows a standard format and level of detail

Best Practices for CloodGroups

Through our implementation, we've developed several best practices for effectively using CloodGroups:

1. Meaningful Grouping

Group related changes into cohesive CloodGroups that represent logical features or components, not just arbitrary collections of files:

{
  "files": [
    "file1.ts",
    "file2.vue",
    "file3.cs",
    "file4.md"
  ],
  "prompt": "Update multiple files"
}

{
  "files": [
    "services/userAuth.ts",
    "components/UserProfile.vue",
    "controllers/UserController.cs",
    "docs/authentication.md"
  ],
  "prompt": "Implement user authentication with JWT tokens, including a login form, backend validation, and token storage"
}

2. Clear, Actionable Prompts

Write prompts that would genuinely allow someone (or an AI) to recreate the implementation:

3. Include Context and Requirements

Make sure prompts include not just what to do, but why it's being done:

4. Consistent Naming Conventions

Use descriptive, consistent filenames for CloodGroups:

Challenges and Solutions

Implementing the CloodGroups system wasn't without challenges. Here's how we addressed some of the key obstacles:

1. Git History Complexity

Challenge: Complex git histories with many small commits made it difficult to generate meaningful CloodGroups.

Solution: We tied CloodGroups to our task tracking system rather than individual commits. Each task has a start and end commit, allowing us to analyze the aggregate changes for a logical unit of work.

-- Track start and end commits for tasks
ALTER TABLE items 
ADD COLUMN start_commit TEXT,
ADD COLUMN end_commit TEXT;

-- Update task commit tracking
UPDATE items 
SET start_commit = '3a4b5c6', end_commit = '7d8e9f0'
WHERE id = 42 AND type = 'task';

2. Maintaining Prompt Quality

Challenge: Automatically generated prompts were sometimes too general or missed key aspects of the implementation.

Solution: We implemented a review step where developers could edit the generated prompt before finalizing the CloodGroup.

#!/bin/bash

# Generate initial CloodGroup
./generate_clood_group.sh $TASK_ID

# Open the generated file for editing
FILENAME=$(sqlite3 /path/to/database.db "SELECT file_path FROM clood_groups WHERE item_id = $TASK_ID ORDER BY id DESC LIMIT 1")
$EDITOR $FILENAME

# Prompt for confirmation
read -p "Are you satisfied with the CloodGroup (y/n)? " confirm
if [ "$confirm" != "y" ]; then
  echo "Please edit the CloodGroup and run this script again."
  exit 1
fi

# Update status in database
sqlite3 /path/to/database.db "UPDATE clood_groups SET status = 'reviewed' WHERE file_path = '$FILENAME'"
echo "CloodGroup finalized at $FILENAME"

3. Large-Scale Changes

Challenge: Very large changes affecting many files were difficult to capture in a single, coherent CloodGroup.

Solution: We implemented a hierarchical approach where large tasks could be split into multiple CloodGroups, each focusing on a specific aspect:

-- Allow CloodGroups to have parent-child relationships
ALTER TABLE clood_groups
ADD COLUMN parent_id INTEGER,
ADD FOREIGN KEY (parent_id) REFERENCES clood_groups (id);

-- Create hierarchical CloodGroups
INSERT INTO clood_groups (item_id, file_path, prompt_content, files_included, parent_id)
VALUES (42, '/path/to/ui-components.json', 'Implement the UI components for the analytics dashboard', '["components/Chart.vue", "components/MetricCard.vue"]', 123);

Case Study: Using CloodGroups with Claude

One of the most powerful applications of CloodGroups is providing better context to Claude when requesting code changes. Here's a real example of how this improved our workflow:

Before CloodGroups

A developer would provide Claude with only the immediate context:

I need to add a new feature to our task progress component that shows the estimated completion time. Here's the current component:

[code for TaskProgress.vue]

Can you update it to include the estimated time to completion?

Without broader context, Claude would make reasonable but potentially misaligned changes based only on the provided file.

After CloodGroups

Now, the developer can provide Claude with the CloodGroup context:

I'm working on implementing a feature described in our CloodGroup:

{
  "files": [
    "Neuro.Client/components/Tasks/TaskProgress.vue",
    "Neuro.Client/composables/useTaskUpdates.ts",
    "Neuro.ServiceModel/Dtos/TaskProgressUpdate.cs"
  ],
  "prompt": "Enhance the task progress tracking system to calculate and display estimated completion time. The system should analyze the progress rate over time and extrapolate a finishing time. The backend will provide progress percentage and elapsed time, while the frontend will calculate and display the time remaining in a user-friendly format (e.g., '2 minutes remaining')."
}

I need to update the TaskProgress.vue component. Here's the current code:

[code for TaskProgress.vue]

Can you implement the estimated completion time display according to the CloodGroup description?

With this context, Claude now understands:

• The broader architecture (backend provides progress and elapsed time)
• The specific calculation needs to happen on the frontend
• The required display format
• Other files that will be involved in the implementation

The result is code that aligns perfectly with our architecture and requirements, requiring fewer revisions and iterations.

Future Directions

While our CloodGroups system has already transformed our documentation process, we have several exciting enhancements planned:

1. Interactive Visualization

We're developing a visual explorer for CloodGroups that will allow developers to:

• See relationships between different CloodGroups
• Navigate the evolution of components over time
• Understand dependencies between features

2. Bidirectional Integration

Currently, CloodGroups are generated from code changes. We're working on reverse functionality:

• Create a CloodGroup first as a specification
• Use Claude to implement based on the CloodGroup
• Track the resulting changes against the original CloodGroup

3. Automated Testing Generation

We plan to extend CloodGroups to automatically generate test cases based on the implementation description:

• Extract testable requirements from prompts
• Generate unit and integration tests
• Create test data based on implementation patterns

4. Natural Language Querying

We're building a system to allow developers to query the codebase using natural language based on CloodGroups:

• "Show me all components related to task progress tracking"
• "What files were changed to implement the authentication system?"
• "Who worked on the real-time notification features?"

Conclusion

CloodGroups represent a fundamental shift in how we approach documentation. By automatically generating structured documentation from git commits, we've transformed documentation from a tedious, often-neglected task into an organic byproduct of the development process.

The real power of CloodGroups lies in their ability to preserve not just what changed, but why it changed. This context is invaluable for onboarding new team members, maintaining complex systems, and providing AI tools like Claude with the information they need to generate appropriate code.

As software systems continue to grow in complexity, approaches like CloodGroups will become increasingly important for maintaining institutional knowledge and ensuring that development teams can work effectively with their codebase. By investing in automated documentation generation now, we're building a foundation for more maintainable, comprehensible software in the future.

For teams considering implementing a similar system, our experience shows that the benefits far outweigh the initial investment. The improved knowledge sharing, reduced documentation overhead, and enhanced AI interactions have made CloodGroups an indispensable part of our development workflow.