diff --git a/.gitignore b/.gitignore index b2be92b..22f5618 100644 --- a/.gitignore +++ b/.gitignore @@ -1 +1,2 @@ result +*.db diff --git a/CLAUDE.md b/CLAUDE.md index 6639a57..3eeb19d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -12,176 +12,157 @@ The first MCP server provides search and query capabilities for NixOS configurat ## Technology Stack -- **Language**: Go 1.25.5 +- **Language**: Go 1.24+ - **Build System**: Nix flakes -- **Databases**: PostgreSQL (primary) and SQLite (lightweight alternative) -- **Protocol**: MCP (Model Context Protocol) - JSON-RPC over stdio +- **Databases**: PostgreSQL and SQLite (both fully supported) +- **Protocol**: MCP (Model Context Protocol) - JSON-RPC over STDIO or HTTP/SSE - **Module Path**: `git.t-juice.club/torjus/labmcp` -## Key Architectural Decisions +## Project Status -1. **Database Support**: Both PostgreSQL and SQLite - - PostgreSQL is preferred for production use (user's preference) - - SQLite provides lightweight alternative for simpler deployments - - Use Go's `database/sql` interface for abstraction +**Complete and maintained** - All core features implemented: +- Full MCP server with 6 tools +- PostgreSQL and SQLite backends with FTS +- NixOS module for deployment +- CLI for manual operations +- Comprehensive test suite -2. **File Storage**: Store nixpkgs file contents in database during indexing - - Better performance for the `get_file` tool - - PostgreSQL handles large text storage well - -3. **Revision Management**: Support multiple indexed nixpkgs revisions - - Store git hash, date, channel name, option count - - Allow querying specific revisions or use defaults - - Default revision: nixos-stable (configurable) - -4. **Indexing Approach**: Part of MCP server, blocking operation (initially) - - Allows Claude to read flake.lock and request indexing - - Can optimize to async later if needed - -5. **Testing**: Aim for >80% test coverage - - Unit tests for all components - - Integration tests for full workflows - - Benchmarks for indexing and query performance - -## MCP Tools to Implement - -### Core Search & Query -1. **`search_options`** - Fuzzy/partial matching search - - Parameters: revision, query, optional filters (type, namespace, hasDefault) - - Returns: matching options with basic metadata - -2. **`get_option`** - Get full details for specific option - - Parameters: revision, option_path, optional depth - - Returns: name, type, default, example, description, file paths - - Default: direct children only (one level deep) - - Includes related/nearby options in same namespace - -3. **`get_file`** - Fetch nixpkgs source file contents - - Parameters: revision, file_path - - Returns: file contents - - Security: validate paths, no traversal, nixpkgs-only - -### Revision Management -4. **`index_revision`** - Index a specific nixpkgs revision - - Parameters: git_hash (full or short) - - Process: fetch nixpkgs, extract options.json, populate DB - - Returns: summary (option count, duration, etc.) - -5. **`list_revisions`** - List indexed revisions - - Returns: git hash, date, channel name, option count - -6. **`delete_revision`** - Prune old/unused revisions - - Parameters: revision identifier - - Returns: confirmation of deletion - -### Channel Support -- Support friendly aliases: `nixos-unstable`, `nixos-24.05`, `nixos-23.11`, etc. -- Can be used in place of git hashes in all tools - -## Database Schema - -**Tables:** - -1. `revisions` - Indexed nixpkgs versions - - id, git_hash (unique), channel_name, commit_date, indexed_at, option_count - -2. `options` - NixOS options with hierarchy support - - id, revision_id (FK), name, parent_path, type, default_value (JSON text), example (JSON text), description, read_only - - parent_path enables efficient "list children" queries (derived from name) - -3. `declarations` - File paths where options are declared - - id, option_id (FK), file_path, line_number - -4. `files` - Cached file contents - - id, revision_id (FK), file_path, extension, content - - Configurable whitelist of extensions (default: .nix, .json, .md, .txt, .toml, .yaml, .yml) - -**Indexes:** -- Full-text search: PostgreSQL (tsvector/GIN), SQLite (FTS5) -- B-tree on (revision_id, name) and (revision_id, parent_path) -- B-tree on (revision_id, file_path) for file lookups - -**Cross-DB Compatibility:** -- JSON stored as TEXT (not JSONB) for SQLite compatibility -- Separate FTS implementations per database engine - -## Repository Structure (Planned) +## Repository Structure ``` labmcp/ ├── cmd/ -│ └── nixos-options/ # MCP server binary -│ └── main.go +│ └── nixos-options/ +│ └── main.go # CLI entry point ├── internal/ -│ ├── mcp/ # MCP protocol implementation -│ │ ├── server.go -│ │ └── types.go -│ ├── database/ # Database abstraction -│ │ ├── interface.go -│ │ ├── postgres.go -│ │ └── sqlite.go -│ └── nixos/ # NixOS options specific logic -│ ├── search.go -│ └── types.go -├── scripts/ -│ └── populate-db.go # Tool to populate database -├── schema/ -│ └── schema.sql # Database schema -├── flake.nix # Nix build configuration +│ ├── database/ +│ │ ├── interface.go # Store interface +│ │ ├── schema.go # Schema versioning +│ │ ├── postgres.go # PostgreSQL implementation +│ │ ├── sqlite.go # SQLite implementation +│ │ └── *_test.go # Database tests +│ ├── mcp/ +│ │ ├── server.go # MCP server core +│ │ ├── handlers.go # Tool implementations +│ │ ├── types.go # Protocol types +│ │ ├── transport.go # Transport interface +│ │ ├── transport_stdio.go # STDIO transport +│ │ ├── transport_http.go # HTTP/SSE transport +│ │ ├── session.go # HTTP session management +│ │ └── *_test.go # MCP tests +│ └── nixos/ +│ ├── indexer.go # Nixpkgs indexing +│ ├── parser.go # options.json parsing +│ ├── types.go # Channel aliases, extensions +│ └── *_test.go # Indexer tests +├── nix/ +│ ├── module.nix # NixOS module +│ └── package.nix # Nix package definition +├── testdata/ +│ └── options-sample.json # Test fixture +├── flake.nix ├── go.mod -├── TODO.md # Detailed task list +├── .mcp.json # MCP client configuration ├── CLAUDE.md # This file -└── README.md +├── README.md +└── TODO.md # Future improvements ``` -## Use Cases +## MCP Tools -**Primary Use Case**: Claude can help users find and understand NixOS options -- "What options are available for nginx?" -- "Show me the services.caddy.* options" -- "What's the default value for services.postgresql.enable?" -- User shares a flake.lock → Claude indexes that nixpkgs version → answers questions about options in that specific version +All tools are implemented and functional: -**Secondary Use Case**: Explore module implementations -- If option description is unclear, fetch the actual module source -- Understand how complex options are structured +| Tool | Description | +|------|-------------| +| `search_options` | Full-text search across option names and descriptions | +| `get_option` | Get full details for a specific option with children | +| `get_file` | Fetch source file contents from indexed nixpkgs | +| `index_revision` | Index a nixpkgs revision (by hash or channel name) | +| `list_revisions` | List all indexed revisions | +| `delete_revision` | Delete an indexed revision | -## Testing Strategy +## Key Implementation Details -- **Unit Tests**: All components with mocks where appropriate -- **Integration Tests**: Full indexing pipeline, MCP tool invocations -- **Benchmarks**: Indexing time, query performance, memory usage -- **Test Fixtures**: Sample options.json, mock repositories -- **Coverage Goal**: >80% on core logic, 100% on database operations +### Database +- Schema versioning with automatic recreation on version mismatch +- Full-text search: SQLite FTS5, PostgreSQL tsvector/GIN +- Path-based queries use LIKE for exact prefix matching +- Batch operations for efficient indexing -## Open Questions +### Indexing +- Uses `nix-build` to evaluate NixOS options from any nixpkgs revision +- File indexing downloads tarball and stores allowed extensions (.nix, .json, .md, etc.) +- File indexing enabled by default (use `--no-files` to skip) +- Skips already-indexed revisions (use `--force` to re-index) -1. Should `index_revision` be blocking or async? (Currently: blocking, optimize later) -2. Should we auto-update channel aliases or manual only? +### Transports +- **STDIO**: Default transport, line-delimited JSON-RPC (for CLI/desktop MCP clients) +- **HTTP**: Streamable HTTP transport with SSE (for web-based MCP clients) + - Session management with cryptographically secure IDs + - Configurable CORS (localhost-only by default) + - Optional TLS support + - SSE keepalive messages (15s default) -## Current Status +### Security +- Revision parameter validated against strict regex to prevent Nix injection +- Path traversal protection using `filepath.Clean()` and `filepath.IsAbs()` +- NixOS module supports `connectionStringFile` for PostgreSQL secrets +- Systemd service runs with extensive hardening options +- HTTP transport hardening: + - Request body size limit (1MB default) + - Server timeouts (read: 30s, write: 30s, idle: 120s, header: 10s) + - Maximum session limit (10,000 default) + - Origin validation for CORS -**Planning phase** - architecture and features defined, ready to begin implementation. +## CLI Commands -## Next Steps - -1. Design and implement database schema -2. Set up project structure (directories, Go modules) -3. Implement database abstraction layer -4. Implement MCP protocol basics -5. Build indexing logic -6. Implement MCP tools -7. Create Nix package in flake.nix -8. Write tests and benchmarks +```bash +nixos-options serve # Run MCP server on STDIO (default) +nixos-options serve --transport http # Run MCP server on HTTP +nixos-options serve --transport http \ + --http-address 0.0.0.0:8080 \ + --allowed-origins https://example.com # HTTP with custom config +nixos-options index # Index a nixpkgs revision +nixos-options index --force # Force re-index existing revision +nixos-options index --no-files # Skip file content indexing +nixos-options list # List indexed revisions +nixos-options search # Search options +nixos-options get