Compare commits

..

26 Commits

Author SHA1 Message Date
5e043e724e chore: add MCP configuration file
Example configuration for running the nixos-options MCP server
via Claude Code or other MCP clients.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 19:14:36 +01:00
cc369e6385 security: add connectionStringFile option for PostgreSQL secrets
The connectionString option stores credentials in the world-readable
Nix store. This adds connectionStringFile as an alternative that reads
the connection string from a file at runtime, compatible with secret
management tools like agenix or sops-nix.

Changes:
- Add database.connectionStringFile option (mutually exclusive with connectionString)
- Read connection string from file at service start when configured
- Add warning to connectionString documentation about Nix store visibility
- Update README with examples for both approaches

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 19:13:20 +01:00
f0adc9efbe security: improve path validation in get_file handler
The previous check only looked for ".." substring, which missed:
- Absolute paths (/etc/passwd)
- URL-encoded traversal patterns
- Paths that clean to traversal (./../../etc)

Now uses filepath.Clean() and filepath.IsAbs() for robust validation:
- Rejects absolute paths
- Cleans paths before checking for traversal
- Uses cleaned path for database lookup

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 19:12:25 +01:00
be1ff4839b security: validate revision parameter to prevent Nix injection
The revision parameter was interpolated directly into a Nix expression,
allowing potential injection of arbitrary Nix code. An attacker could
craft a revision string like:
  "; builtins.readFile /etc/passwd; "

This adds ValidateRevision() which ensures revisions only contain safe
characters (alphanumeric, hyphens, underscores, dots) and are at most
64 characters long.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 19:10:31 +01:00
5f0445e749 docs: add nix develop note to CLAUDE.md
Go commands should be run via nix develop -c to ensure the proper
build environment with all dependencies (like gcc for cgo).

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 19:01:52 +01:00
730f2d7610 feat: skip already-indexed revisions, add --force flag
When indexing a revision that already exists, the indexer now returns
early with information about the existing revision instead of re-indexing.
Use the --force flag to re-index an existing revision.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 18:59:44 +01:00
ae6a4d6cf9 feat: add --version flag to CLI
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 18:55:44 +01:00
11c300c4e7 docs: update TODO.md with future improvements
Replace completed planning document with actionable improvement ideas.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 18:53:55 +01:00
8627bfbe0a feat: add MCP server instructions for flake.lock indexing
Provides guidance to AI assistants on how to index the nixpkgs
revision from a project's flake.lock file, ensuring option
documentation matches the project's actual nixpkgs version.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 18:50:51 +01:00
452b0fda86 feat: enable file indexing by default
File indexing only adds ~3 seconds to the indexing process, so enable
it by default to make the get_file tool work out of the box.

- MCP index_revision tool now indexes files automatically
- CLI flag changed from --files to --no-files (opt-out)
- Update README examples

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 18:45:50 +01:00
3ba85691a8 test: improve file indexing benchmarks
- BenchmarkIndexRevisionWithFiles now reports separate timing for
  options indexing (options_ms) and file indexing (files_ms)
- Add BenchmarkIndexFilesOnly to measure file indexing in isolation

Run with:
  go test -bench=BenchmarkIndexFilesOnly -benchtime=1x -timeout=60m ./internal/nixos/...

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 18:42:02 +01:00
23076fa112 refactor: move package definition to nix/package.nix
Keeps flake.nix cleaner by extracting the package definition.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 18:29:37 +01:00
e2c006cb9f docs: add NixOS module documentation to README
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 18:27:11 +01:00
43ffc234ac feat: add NixOS module for nixos-options-mcp service
Module provides:
- services.nixos-options-mcp.enable - Enable the service
- services.nixos-options-mcp.package - Package to use
- services.nixos-options-mcp.database.type - sqlite or postgres
- services.nixos-options-mcp.database.name - SQLite filename
- services.nixos-options-mcp.database.connectionString - PostgreSQL URL
- services.nixos-options-mcp.indexOnStart - Revisions to index on start
- services.nixos-options-mcp.user/group - Service user/group
- services.nixos-options-mcp.dataDir - Data directory

Includes systemd hardening options.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 18:26:29 +01:00
88e8a55347 fix: improve search to prioritize path-based matching
When searching for option paths like "services.nginx", use name-based
LIKE matching instead of full-text search. This ensures the results
are options that start with the query, not random options that mention
the term somewhere in their description.

- Path queries (containing dots): use LIKE for name prefix matching
- Text queries (no dots): use FTS for full-text search on name+description

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 18:23:50 +01:00
ec0eba4bef fix: escape FTS5 queries to handle special characters
Wrap search queries in double quotes for FTS5 literal matching.
This prevents dots, colons, and other special characters from
being interpreted as FTS5 operators.

Fixes: "fts5: syntax error near '.'" when searching for option
paths like "services.nginx".

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 18:09:32 +01:00
d82990fbfa test: add test for search with special characters
Tests searching with dots, colons, hyphens, and parentheses.
Currently fails on SQLite due to FTS5 syntax interpretation.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 18:08:55 +01:00
9352fd1f6e docs: update README with usage instructions
- Add installation instructions (nix flakes, go install)
- Add MCP server configuration example
- Add CLI examples for all commands
- Document environment variables and database connection strings
- List available MCP tools

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 17:58:02 +01:00
849ff38597 feat: wrap nixos-options binary with nix in PATH
The indexer requires nix-build to evaluate NixOS options.
Use makeWrapper to add nix to the binary's PATH.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 17:55:34 +01:00
f18a7e2626 test: add indexer benchmark and integration test
- BenchmarkIndexRevision: benchmark full nixpkgs indexing
- BenchmarkIndexRevisionWithFiles: benchmark with file content storage
- TestIndexRevision: integration test for indexer
- Uses nixpkgs revision from flake.lock (e6eae2ee...)
- Skips if nix-build not available or in short mode

Run with: go test -bench=BenchmarkIndexRevision -benchtime=1x -timeout=30m ./internal/nixos/...

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 17:55:34 +01:00
610dc7bd61 chore: add CLAUDE.md and gitignore
- Add project context file for Claude
- Ignore nix result symlink

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 17:55:34 +01:00
939abc8d8e test: MCP server tests, benchmarks, and nix build fix
- Add MCP server protocol tests (initialize, tools/list, errors)
- Add database benchmarks (batch inserts, search, children)
- Add sample options.json test fixture
- Fix flake.nix vendor hash for nix build

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 17:54:42 +01:00
f7112d4459 feat: CLI integration with database and MCP server
- Wire up all CLI commands to database operations
- Add 'get' command for single option details
- Add '--files' flag to 'index' for file content indexing
- Support postgres:// and sqlite:// connection strings
- Default to SQLite database file

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 17:54:42 +01:00
0b0ada3ccd feat: MCP tools and nixpkgs indexer
- Add options.json parser with mdDoc support
- Add nixpkgs indexer using nix-build
- Implement all MCP tool handlers:
  - search_options: Full-text search with filters
  - get_option: Option details with children
  - get_file: Fetch file contents
  - index_revision: Build and index options
  - list_revisions: Show indexed versions
  - delete_revision: Remove indexed data
- Add parser tests

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 17:54:42 +01:00
93245c1439 test: database layer tests
- Add comprehensive test suite for Store interface
- Test schema initialization, revisions, options, search, declarations, files
- SQLite tests use in-memory database for speed
- PostgreSQL tests require TEST_POSTGRES_CONN environment variable

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 17:54:42 +01:00
6326b3a3c1 feat: project structure and nix build setup
- Add CLI entry point with urfave/cli/v2 (serve, index, list, search commands)
- Add database interface and implementations for PostgreSQL and SQLite
- Add schema versioning with automatic recreation on version mismatch
- Add MCP protocol types and server scaffold
- Add NixOS option types
- Configure flake.nix with devShell and buildGoModule package

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-03 17:54:42 +01:00
30 changed files with 5433 additions and 226 deletions

1
.gitignore vendored Normal file
View File

@@ -0,0 +1 @@
result

16
.mcp.json Normal file
View File

@@ -0,0 +1,16 @@
{
"mcpServers": {
"nixos-options": {
"command": "nix",
"args": [
"run",
".",
"--",
"serve"
],
"env": {
"NIXOS_OPTIONS_DATABASE": "sqlite://:memory:"
}
}
}
}

View File

@@ -78,18 +78,32 @@ The first MCP server provides search and query capabilities for NixOS configurat
- Support friendly aliases: `nixos-unstable`, `nixos-24.05`, `nixos-23.11`, etc. - Support friendly aliases: `nixos-unstable`, `nixos-24.05`, `nixos-23.11`, etc.
- Can be used in place of git hashes in all tools - Can be used in place of git hashes in all tools
## Database Schema (Planned) ## Database Schema
**Tables:** **Tables:**
- `revisions` - nixpkgs git hash, date, channel name, metadata
- `options` - per revision: name, type, default, example, description 1. `revisions` - Indexed nixpkgs versions
- `declarations` - file paths where options are declared - id, git_hash (unique), channel_name, commit_date, indexed_at, option_count
- `files` - cached nixpkgs file contents for `get_file` tool
2. `options` - NixOS options with hierarchy support
- id, revision_id (FK), name, parent_path, type, default_value (JSON text), example (JSON text), description, read_only
- parent_path enables efficient "list children" queries (derived from name)
3. `declarations` - File paths where options are declared
- id, option_id (FK), file_path, line_number
4. `files` - Cached file contents
- id, revision_id (FK), file_path, extension, content
- Configurable whitelist of extensions (default: .nix, .json, .md, .txt, .toml, .yaml, .yml)
**Indexes:** **Indexes:**
- Full-text search on option names and descriptions - Full-text search: PostgreSQL (tsvector/GIN), SQLite (FTS5)
- B-tree indexes on revision + option name - B-tree on (revision_id, name) and (revision_id, parent_path)
- Namespace prefix indexes for category filtering - B-tree on (revision_id, file_path) for file lookups
**Cross-DB Compatibility:**
- JSON stored as TEXT (not JSONB) for SQLite compatibility
- Separate FTS implementations per database engine
## Repository Structure (Planned) ## Repository Structure (Planned)
@@ -168,3 +182,6 @@ labmcp/
- Nix flake must provide importable packages for other repos - Nix flake must provide importable packages for other repos
- Use `database/sql` interface for database abstraction - Use `database/sql` interface for database abstraction
- File paths in responses should use format `path/to/file.go:123` - File paths in responses should use format `path/to/file.go:123`
- **Always run `go fmt ./...` before committing Go code**
- **Run Go commands using `nix develop -c`** (e.g., `nix develop -c go test ./...`) to ensure proper build environment with all dependencies
- **Use `nix run` to run binaries** instead of `go build` followed by running the binary (e.g., `nix run .#nixos-options -- serve`)

244
README.md
View File

@@ -4,20 +4,242 @@ A collection of Model Context Protocol (MCP) servers written in Go.
## NixOS Options MCP Server ## NixOS Options MCP Server
Search and query NixOS configuration options across multiple nixpkgs revisions. Search and query NixOS configuration options across multiple nixpkgs revisions. Designed to help Claude (and other MCP clients) answer questions about NixOS configuration.
**Features:** ### Features
- Search options with fuzzy matching
- Query specific options with full metadata - Full-text search across option names and descriptions
- Index multiple nixpkgs revisions - Query specific options with type, default, example, and declarations
- Index multiple nixpkgs revisions (by git hash or channel name)
- Fetch nixpkgs module source files - Fetch nixpkgs module source files
- Support for PostgreSQL and SQLite - Support for PostgreSQL and SQLite backends
## Status ## Installation
🚧 **In Development** - Not yet functional ### Using Nix Flakes
## Documentation ```bash
# Build the package
nix build github:torjus/labmcp
- See [TODO.md](TODO.md) for implementation progress # Or run directly
- See [CLAUDE.md](CLAUDE.md) for architecture and design decisions nix run github:torjus/labmcp -- --help
```
### From Source
```bash
go install git.t-juice.club/torjus/labmcp/cmd/nixos-options@latest
```
## Usage
### As MCP Server
Configure in your MCP client (e.g., Claude Desktop):
```json
{
"mcpServers": {
"nixos-options": {
"command": "nixos-options",
"args": ["serve"],
"env": {
"NIXOS_OPTIONS_DATABASE": "sqlite:///path/to/nixos-options.db"
}
}
}
}
```
Then start the server:
```bash
nixos-options serve
```
### CLI Examples
**Index a nixpkgs revision:**
```bash
# Index by channel name (includes file contents by default)
nixos-options index nixos-unstable
# Index by git hash
nixos-options index e6eae2ee2110f3d31110d5c222cd395303343b08
# Index without file contents (faster, disables get_file tool)
nixos-options index --no-files nixos-unstable
```
**List indexed revisions:**
```bash
nixos-options list
```
**Search for options:**
```bash
# Basic search
nixos-options search nginx
# Limit results
nixos-options search -n 10 postgresql
# Search in specific revision
nixos-options search -r nixos-unstable firewall
```
**Get option details:**
```bash
nixos-options get services.nginx.enable
nixos-options get services.postgresql.package
```
**Delete an indexed revision:**
```bash
nixos-options delete nixos-23.11
```
## Configuration
### Environment Variables
| Variable | Description | Default |
|----------|-------------|---------|
| `NIXOS_OPTIONS_DATABASE` | Database connection string | `sqlite://nixos-options.db` |
### Database Connection Strings
**SQLite:**
```bash
export NIXOS_OPTIONS_DATABASE="sqlite:///path/to/database.db"
export NIXOS_OPTIONS_DATABASE="sqlite://:memory:" # In-memory
```
**PostgreSQL:**
```bash
export NIXOS_OPTIONS_DATABASE="postgres://user:pass@localhost/nixos_options?sslmode=disable"
```
### Command-Line Flags
The database can also be specified via the `-d` or `--database` flag:
```bash
nixos-options -d "postgres://localhost/nixos" serve
nixos-options -d "sqlite://my.db" index nixos-unstable
```
## MCP Tools
When running as an MCP server, the following tools are available:
| Tool | Description |
|------|-------------|
| `search_options` | Search for options by name or description |
| `get_option` | Get full details for a specific option |
| `get_file` | Fetch source file contents from nixpkgs |
| `index_revision` | Index a nixpkgs revision |
| `list_revisions` | List all indexed revisions |
| `delete_revision` | Delete an indexed revision |
## NixOS Module
A NixOS module is provided for running the MCP server as a systemd service.
```nix
{
inputs.labmcp.url = "github:torjus/labmcp";
outputs = { self, nixpkgs, labmcp }: {
nixosConfigurations.myhost = nixpkgs.lib.nixosSystem {
system = "x86_64-linux";
modules = [
labmcp.nixosModules.nixos-options-mcp
{
services.nixos-options-mcp = {
enable = true;
indexOnStart = [ "nixos-unstable" ];
};
}
];
};
};
}
```
### Module Options
| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `enable` | bool | `false` | Enable the service |
| `package` | package | from flake | Package to use |
| `database.type` | enum | `"sqlite"` | `"sqlite"` or `"postgres"` |
| `database.name` | string | `"nixos-options.db"` | SQLite database filename |
| `database.connectionString` | string | `""` | PostgreSQL connection URL (stored in Nix store) |
| `database.connectionStringFile` | path | `null` | Path to file with PostgreSQL connection URL (recommended for secrets) |
| `indexOnStart` | list of string | `[]` | Revisions to index on service start |
| `user` | string | `"nixos-options-mcp"` | User to run the service as |
| `group` | string | `"nixos-options-mcp"` | Group to run the service as |
| `dataDir` | path | `/var/lib/nixos-options-mcp` | Directory for data storage |
### PostgreSQL Example
Using `connectionString` (stored in Nix store - suitable for testing or non-sensitive setups):
```nix
{
services.nixos-options-mcp = {
enable = true;
database = {
type = "postgres";
connectionString = "postgres://nixos:nixos@localhost/nixos_options?sslmode=disable";
};
indexOnStart = [ "nixos-unstable" "nixos-24.11" ];
};
}
```
Using `connectionStringFile` (recommended for production with sensitive credentials):
```nix
{
services.nixos-options-mcp = {
enable = true;
database = {
type = "postgres";
# File contains: postgres://user:secret@localhost/nixos_options?sslmode=disable
connectionStringFile = "/run/secrets/nixos-options-db";
};
indexOnStart = [ "nixos-unstable" ];
};
# Example with agenix or sops-nix for secret management
# age.secrets.nixos-options-db.file = ./secrets/nixos-options-db.age;
}
```
## Development
```bash
# Enter development shell
nix develop
# Run tests
go test ./...
# Run benchmarks
go test -bench=. ./internal/database/...
# Build
go build ./cmd/nixos-options
```
## License
MIT

213
TODO.md
View File

@@ -1,207 +1,18 @@
# LabMCP - MCP Server Collection # TODO - Future Improvements
## Project Summary ## Usability
This repository hosts a collection of Model Context Protocol (MCP) servers designed to extend Claude's capabilities with custom tools. The project is structured to be generic and extensible, allowing multiple MCP servers to be added over time. - [ ] Progress reporting during indexing ("Fetching nixpkgs... Parsing options... Indexing files...")
- [ ] Add `search_files` MCP tool - search for files by path pattern (e.g., find all nginx-related modules)
**Technology Stack:** ## Robustness
- Language: Go
- Build System: Nix/NixOS
- Deployment: Nix flake providing importable packages
## Initial MCP Server: NixOS Options Search - [ ] PostgreSQL integration tests with testcontainers (currently skipped without manual DB setup)
- [ ] Graceful handling of concurrent indexing (what happens if two clients index the same revision?)
The first MCP server will provide search capabilities for NixOS options. This addresses the challenge of incomplete or hard-to-find documentation in the Nix ecosystem by allowing Claude to query available NixOS configuration options directly. ## Nice to Have
## TODO - [ ] Option history/diff - compare options between two revisions ("what changed in services.nginx between 24.05 and 24.11?")
- [ ] Auto-cleanup - prune old revisions after N days or keep only N most recent
### Planning & Architecture - [ ] Man page generation
- [ ] Decide on NixOS options data source - [ ] Shell completions (bash, zsh, fish)
- Options: `nixos-option`, JSON exports, search.nixos.org API, etc.
- [ ] Define MCP server protocol implementation approach
- How to structure the Go MCP server
- What tools/resources to expose via MCP
- [ ] Plan repository structure
- Monorepo with multiple servers?
- Shared libraries/utilities?
- How to organize flake outputs
### NixOS Options MCP Server
- [ ] Implement basic MCP server in Go
- [ ] Add NixOS options search functionality
- [ ] Define search capabilities (by name, description, type, etc.)
- [ ] Handle option metadata (type, default, example, description)
### Nix/Build Configuration
- [ ] Set up flake.nix with proper structure
- [ ] Create buildable package for the MCP server
- [ ] Ensure package is importable from other repositories
- [ ] Add development shell for local testing
### Testing & Documentation
- [ ] Unit tests for all major components
- Database operations (both PostgreSQL and SQLite)
- MCP protocol handling
- NixOS options parsing
- Search functionality
- File retrieval
- [ ] Integration tests
- End-to-end indexing workflow
- MCP tool invocations
- Multi-revision scenarios
- [ ] Benchmarks
- Indexing performance (time to index a full nixpkgs revision)
- Search query performance
- Database query optimization
- Memory usage during indexing
- [ ] Test fixtures
- Sample options.json files
- Mock nixpkgs repositories for testing
- Test databases with known data
- [ ] Test MCP server with Claude (real-world usage)
- [ ] Document usage and installation
- [ ] Add examples of useful queries
## Decisions Made
1. **Data Source**: Use NixOS options.json exports, pre-processed and indexed
2. **Database**: Support both PostgreSQL (primary) and SQLite (lightweight option)
3. **Language**: Go with database/sql abstraction layer
4. **Build**: Nix flake with importable packages
## MCP Tools / Features
### Core Search & Query
1. **`search_options`** - Fuzzy/partial matching search
- Search by name (e.g., "nginx" matches "services.nginx.*")
- Full-text search in descriptions
- Requires: revision (git hash or alias), query string
- Optional filters:
- Option type (boolean, string, path, package, etc.)
- Namespace/category (services.*, programs.*, etc.)
- Has default value (true/false)
2. **`get_option`** - Get full details for specific option
- Returns: name, type, default, example, description, declarations (file paths)
- Limit depth to prevent returning thousands of sub-options
- Default: direct children only (one level deep)
- Optional: `depth` parameter for recursive queries
- Show related/nearby options in same namespace
- Requires: revision, option path
3. **`get_file`** - Fetch nixpkgs source file contents
- Example: `modules/services/web-servers/caddy/default.nix`
- Useful when option descriptions are unclear
- Security: Path validation, no traversal, nixpkgs-only
- Requires: revision, file path
### Revision Management
4. **`index_revision`** - Index a specific nixpkgs revision
- Takes: git hash (full or short)
- Fetches nixpkgs, extracts options.json, populates database
- Stores metadata: hash, date, channel name (if known)
- May be slow - consider async operation
5. **`list_revisions`** - List indexed revisions
- Returns: git hash, date, channel name, option count
- Helps users see what's available before querying
6. **`delete_revision`** - Prune old/unused revisions
- Remove revision and all associated data from database
- Useful for DB maintenance and disk space management
### Channel/Tag Support
- Support friendly aliases: `nixos-unstable`, `nixos-24.05`, `nixos-23.11`
- Can auto-update these periodically or on-demand
- Allow searching by channel name instead of git hash
## Database Schema
**Tables:**
- `revisions` - nixpkgs git hash, date, channel name, metadata
- `options` - per revision: name, type, default, example, description
- `declarations` - file paths where options are declared
- `files` (optional) - cache of nixpkgs file contents for faster access
**Indexes:**
- Full-text search on option names and descriptions
- B-tree indexes on revision + option name for fast lookups
- Namespace prefix indexes for category filtering
## Implementation Considerations
**Result Limiting:**
- When getting options, return direct children only by default
- If result set exceeds threshold, return summary + suggestion to narrow query
- Use pagination for large search results
**File Fetching Strategy:**
- Option 1: Keep shallow git clones of indexed revisions
- Option 2: Store file contents in DB during indexing (faster, more storage)
- Leaning toward Option 2 for better MCP performance
**Revision Indexing:**
- Potentially slow operation (download + parse + index)
- Consider: Separate admin tool vs MCP tool vs async queue
- For initial version: Blocking operation, can optimize later
**Default Behavior:**
- Should search require specifying revision?
- Could default to "latest indexed" or "nixos-unstable"
- Make configurable
## Design Decisions - Finalized
1. **File Storage**: Store file contents in database during indexing
- Better performance for `get_file` tool
- PostgreSQL handles this well
2. **Default Revision**: Default to configured channel (nixos-stable recommended)
- Conservative approach - stable options likely exist in unstable too
- Configurable via environment variable or config file
3. **Index as MCP Tool**: `index_revision` is part of MCP server
- Allows Claude to read flake.nix/flake.lock and request indexing
- Use case: "Index the nixpkgs version used by this flake"
- To decide: Blocking vs async (start with blocking, optimize later)
4. **Testing & Quality**
- Aim for good test coverage across all components
- Write tests alongside implementation (TDD where practical)
- Include benchmarks to measure indexing performance
- Use test fixtures for reproducible testing
- Test both PostgreSQL and SQLite implementations
## Testing Strategy
**Unit Testing:**
- Database layer: Test with both real databases and mocks
- Use testcontainers or docker-compose for PostgreSQL tests
- SQLite tests can use in-memory databases
- Mock external dependencies (git operations, network calls)
**Integration Testing:**
- Use small, curated test fixtures (subset of real options.json)
- Test full indexing pipeline with known data
- Verify search results match expectations
- Test error handling (malformed data, missing revisions, etc.)
**Benchmarking:**
- Measure time to index full nixpkgs revision
- Track query performance as database grows
- Memory profiling during indexing
- Compare PostgreSQL vs SQLite performance
- Use Go's built-in benchmarking: `go test -bench=.`
**Test Coverage Goals:**
- Aim for >80% coverage on core logic
- 100% coverage on database operations
- Focus on critical paths and error handling
## Questions to Resolve
1. **Revision Auto-Update**: Should we auto-update channel aliases, or manual only?
2. **Indexing Behavior**: Should `index_revision` be blocking or async?
- Blocking: Simpler to implement, Claude waits for completion
- Async: Better UX for slow operations, need status checking
- Recommendation: Start with blocking, add async later if needed

457
cmd/nixos-options/main.go Normal file
View File

@@ -0,0 +1,457 @@
package main
import (
"context"
"fmt"
"log"
"os"
"strings"
"github.com/urfave/cli/v2"
"git.t-juice.club/torjus/labmcp/internal/database"
"git.t-juice.club/torjus/labmcp/internal/mcp"
"git.t-juice.club/torjus/labmcp/internal/nixos"
)
const (
defaultDatabase = "sqlite://nixos-options.db"
version = "0.1.0"
)
func main() {
app := &cli.App{
Name: "nixos-options",
Usage: "MCP server for NixOS options search and query",
Version: version,
Flags: []cli.Flag{
&cli.StringFlag{
Name: "database",
Aliases: []string{"d"},
Usage: "Database connection string (postgres://... or sqlite://...)",
EnvVars: []string{"NIXOS_OPTIONS_DATABASE"},
Value: defaultDatabase,
},
},
Commands: []*cli.Command{
{
Name: "serve",
Usage: "Run MCP server (stdio)",
Action: func(c *cli.Context) error {
return runServe(c)
},
},
{
Name: "index",
Usage: "Index a nixpkgs revision",
ArgsUsage: "<revision>",
Flags: []cli.Flag{
&cli.BoolFlag{
Name: "no-files",
Usage: "Skip indexing file contents (faster, disables get_file tool)",
},
&cli.BoolFlag{
Name: "force",
Aliases: []string{"f"},
Usage: "Force re-indexing even if revision already exists",
},
},
Action: func(c *cli.Context) error {
if c.NArg() < 1 {
return fmt.Errorf("revision argument required")
}
return runIndex(c, c.Args().First(), !c.Bool("no-files"), c.Bool("force"))
},
},
{
Name: "list",
Usage: "List indexed revisions",
Action: func(c *cli.Context) error {
return runList(c)
},
},
{
Name: "search",
Usage: "Search for options",
ArgsUsage: "<query>",
Flags: []cli.Flag{
&cli.StringFlag{
Name: "revision",
Aliases: []string{"r"},
Usage: "Revision to search (default: most recent)",
},
&cli.IntFlag{
Name: "limit",
Aliases: []string{"n"},
Usage: "Maximum number of results",
Value: 20,
},
},
Action: func(c *cli.Context) error {
if c.NArg() < 1 {
return fmt.Errorf("query argument required")
}
return runSearch(c, c.Args().First())
},
},
{
Name: "get",
Usage: "Get details for a specific option",
ArgsUsage: "<option-name>",
Flags: []cli.Flag{
&cli.StringFlag{
Name: "revision",
Aliases: []string{"r"},
Usage: "Revision to search (default: most recent)",
},
},
Action: func(c *cli.Context) error {
if c.NArg() < 1 {
return fmt.Errorf("option name required")
}
return runGet(c, c.Args().First())
},
},
{
Name: "delete",
Usage: "Delete an indexed revision",
ArgsUsage: "<revision>",
Action: func(c *cli.Context) error {
if c.NArg() < 1 {
return fmt.Errorf("revision argument required")
}
return runDelete(c, c.Args().First())
},
},
},
}
if err := app.Run(os.Args); err != nil {
log.Fatal(err)
}
}
// openStore opens a database store based on the connection string.
func openStore(connStr string) (database.Store, error) {
if strings.HasPrefix(connStr, "sqlite://") {
path := strings.TrimPrefix(connStr, "sqlite://")
return database.NewSQLiteStore(path)
}
if strings.HasPrefix(connStr, "postgres://") || strings.HasPrefix(connStr, "postgresql://") {
return database.NewPostgresStore(connStr)
}
// Default to SQLite with the connection string as path
return database.NewSQLiteStore(connStr)
}
func runServe(c *cli.Context) error {
ctx := context.Background()
store, err := openStore(c.String("database"))
if err != nil {
return fmt.Errorf("failed to open database: %w", err)
}
defer store.Close()
if err := store.Initialize(ctx); err != nil {
return fmt.Errorf("failed to initialize database: %w", err)
}
logger := log.New(os.Stderr, "[mcp] ", log.LstdFlags)
server := mcp.NewServer(store, logger)
indexer := nixos.NewIndexer(store)
server.RegisterHandlers(indexer)
logger.Println("Starting MCP server on stdio...")
return server.Run(ctx, os.Stdin, os.Stdout)
}
func runIndex(c *cli.Context, revision string, indexFiles bool, force bool) error {
ctx := context.Background()
store, err := openStore(c.String("database"))
if err != nil {
return fmt.Errorf("failed to open database: %w", err)
}
defer store.Close()
if err := store.Initialize(ctx); err != nil {
return fmt.Errorf("failed to initialize database: %w", err)
}
indexer := nixos.NewIndexer(store)
fmt.Printf("Indexing revision: %s\n", revision)
var result *nixos.IndexResult
if force {
result, err = indexer.ReindexRevision(ctx, revision)
} else {
result, err = indexer.IndexRevision(ctx, revision)
}
if err != nil {
return fmt.Errorf("indexing failed: %w", err)
}
if result.AlreadyIndexed {
fmt.Printf("Revision already indexed (%d options). Use --force to re-index.\n", result.OptionCount)
return nil
}
fmt.Printf("Indexed %d options in %s\n", result.OptionCount, result.Duration)
fmt.Printf("Git hash: %s\n", result.Revision.GitHash)
if result.Revision.ChannelName != "" {
fmt.Printf("Channel: %s\n", result.Revision.ChannelName)
}
if indexFiles {
fmt.Println("Indexing files...")
fileCount, err := indexer.IndexFiles(ctx, result.Revision.ID, result.Revision.GitHash)
if err != nil {
return fmt.Errorf("file indexing failed: %w", err)
}
fmt.Printf("Indexed %d files\n", fileCount)
}
return nil
}
func runList(c *cli.Context) error {
ctx := context.Background()
store, err := openStore(c.String("database"))
if err != nil {
return fmt.Errorf("failed to open database: %w", err)
}
defer store.Close()
if err := store.Initialize(ctx); err != nil {
return fmt.Errorf("failed to initialize database: %w", err)
}
revisions, err := store.ListRevisions(ctx)
if err != nil {
return fmt.Errorf("failed to list revisions: %w", err)
}
if len(revisions) == 0 {
fmt.Println("No revisions indexed.")
fmt.Println("Use 'nixos-options index <revision>' to index a nixpkgs version.")
return nil
}
fmt.Printf("Indexed revisions (%d):\n\n", len(revisions))
for _, rev := range revisions {
fmt.Printf(" %s", rev.GitHash[:12])
if rev.ChannelName != "" {
fmt.Printf(" (%s)", rev.ChannelName)
}
fmt.Printf("\n Options: %d, Indexed: %s\n",
rev.OptionCount, rev.IndexedAt.Format("2006-01-02 15:04"))
}
return nil
}
func runSearch(c *cli.Context, query string) error {
ctx := context.Background()
store, err := openStore(c.String("database"))
if err != nil {
return fmt.Errorf("failed to open database: %w", err)
}
defer store.Close()
if err := store.Initialize(ctx); err != nil {
return fmt.Errorf("failed to initialize database: %w", err)
}
// Find revision
var rev *database.Revision
revisionArg := c.String("revision")
if revisionArg != "" {
rev, err = store.GetRevision(ctx, revisionArg)
if err != nil {
return fmt.Errorf("failed to get revision: %w", err)
}
if rev == nil {
rev, err = store.GetRevisionByChannel(ctx, revisionArg)
if err != nil {
return fmt.Errorf("failed to get revision: %w", err)
}
}
} else {
revisions, err := store.ListRevisions(ctx)
if err != nil {
return fmt.Errorf("failed to list revisions: %w", err)
}
if len(revisions) > 0 {
rev = revisions[0]
}
}
if rev == nil {
return fmt.Errorf("no indexed revision found")
}
filters := database.SearchFilters{
Limit: c.Int("limit"),
}
options, err := store.SearchOptions(ctx, rev.ID, query, filters)
if err != nil {
return fmt.Errorf("search failed: %w", err)
}
if len(options) == 0 {
fmt.Printf("No options found matching '%s'\n", query)
return nil
}
fmt.Printf("Found %d options matching '%s':\n\n", len(options), query)
for _, opt := range options {
fmt.Printf(" %s\n", opt.Name)
fmt.Printf(" Type: %s\n", opt.Type)
if opt.Description != "" {
desc := opt.Description
if len(desc) > 100 {
desc = desc[:100] + "..."
}
fmt.Printf(" %s\n", desc)
}
fmt.Println()
}
return nil
}
func runGet(c *cli.Context, name string) error {
ctx := context.Background()
store, err := openStore(c.String("database"))
if err != nil {
return fmt.Errorf("failed to open database: %w", err)
}
defer store.Close()
if err := store.Initialize(ctx); err != nil {
return fmt.Errorf("failed to initialize database: %w", err)
}
// Find revision
var rev *database.Revision
revisionArg := c.String("revision")
if revisionArg != "" {
rev, err = store.GetRevision(ctx, revisionArg)
if err != nil {
return fmt.Errorf("failed to get revision: %w", err)
}
if rev == nil {
rev, err = store.GetRevisionByChannel(ctx, revisionArg)
if err != nil {
return fmt.Errorf("failed to get revision: %w", err)
}
}
} else {
revisions, err := store.ListRevisions(ctx)
if err != nil {
return fmt.Errorf("failed to list revisions: %w", err)
}
if len(revisions) > 0 {
rev = revisions[0]
}
}
if rev == nil {
return fmt.Errorf("no indexed revision found")
}
opt, err := store.GetOption(ctx, rev.ID, name)
if err != nil {
return fmt.Errorf("failed to get option: %w", err)
}
if opt == nil {
return fmt.Errorf("option '%s' not found", name)
}
fmt.Printf("%s\n", opt.Name)
fmt.Printf(" Type: %s\n", opt.Type)
if opt.Description != "" {
fmt.Printf(" Description: %s\n", opt.Description)
}
if opt.DefaultValue != "" && opt.DefaultValue != "null" {
fmt.Printf(" Default: %s\n", opt.DefaultValue)
}
if opt.Example != "" && opt.Example != "null" {
fmt.Printf(" Example: %s\n", opt.Example)
}
if opt.ReadOnly {
fmt.Println(" Read-only: yes")
}
// Get declarations
declarations, err := store.GetDeclarations(ctx, opt.ID)
if err == nil && len(declarations) > 0 {
fmt.Println(" Declared in:")
for _, decl := range declarations {
if decl.Line > 0 {
fmt.Printf(" - %s:%d\n", decl.FilePath, decl.Line)
} else {
fmt.Printf(" - %s\n", decl.FilePath)
}
}
}
// Get children
children, err := store.GetChildren(ctx, rev.ID, opt.Name)
if err == nil && len(children) > 0 {
fmt.Println(" Sub-options:")
for _, child := range children {
shortName := child.Name
if strings.HasPrefix(child.Name, opt.Name+".") {
shortName = child.Name[len(opt.Name)+1:]
}
fmt.Printf(" - %s (%s)\n", shortName, child.Type)
}
}
return nil
}
func runDelete(c *cli.Context, revision string) error {
ctx := context.Background()
store, err := openStore(c.String("database"))
if err != nil {
return fmt.Errorf("failed to open database: %w", err)
}
defer store.Close()
if err := store.Initialize(ctx); err != nil {
return fmt.Errorf("failed to initialize database: %w", err)
}
// Find revision
rev, err := store.GetRevision(ctx, revision)
if err != nil {
return fmt.Errorf("failed to get revision: %w", err)
}
if rev == nil {
rev, err = store.GetRevisionByChannel(ctx, revision)
if err != nil {
return fmt.Errorf("failed to get revision: %w", err)
}
}
if rev == nil {
return fmt.Errorf("revision '%s' not found", revision)
}
if err := store.DeleteRevision(ctx, rev.ID); err != nil {
return fmt.Errorf("failed to delete revision: %w", err)
}
fmt.Printf("Deleted revision %s\n", rev.GitHash)
return nil
}

27
flake.lock generated Normal file
View File

@@ -0,0 +1,27 @@
{
"nodes": {
"nixpkgs": {
"locked": {
"lastModified": 1770115704,
"narHash": "sha256-KHFT9UWOF2yRPlAnSXQJh6uVcgNcWlFqqiAZ7OVlHNc=",
"owner": "nixos",
"repo": "nixpkgs",
"rev": "e6eae2ee2110f3d31110d5c222cd395303343b08",
"type": "github"
},
"original": {
"owner": "nixos",
"ref": "nixos-unstable",
"repo": "nixpkgs",
"type": "github"
}
},
"root": {
"inputs": {
"nixpkgs": "nixpkgs"
}
}
},
"root": "root",
"version": 7
}

View File

@@ -1,15 +1,56 @@
{ {
description = "A very basic flake"; description = "LabMCP - Collection of MCP servers";
inputs = { inputs = {
nixpkgs.url = "github:nixos/nixpkgs?ref=nixos-unstable"; nixpkgs.url = "github:nixos/nixpkgs?ref=nixos-unstable";
}; };
outputs = { self, nixpkgs }: { outputs = { self, nixpkgs }:
let
lib = nixpkgs.lib;
supportedSystems = [ "x86_64-linux" "aarch64-linux" "x86_64-darwin" "aarch64-darwin" ];
forAllSystems = lib.genAttrs supportedSystems;
pkgsFor = system: nixpkgs.legacyPackages.${system};
in
{
packages = forAllSystems (system:
let
pkgs = pkgsFor system;
in
{
nixos-options = pkgs.callPackage ./nix/package.nix { src = ./.; };
default = self.packages.${system}.nixos-options;
});
packages.x86_64-linux.hello = nixpkgs.legacyPackages.x86_64-linux.hello; devShells = forAllSystems (system:
let
pkgs = pkgsFor system;
in
{
default = pkgs.mkShell {
buildInputs = with pkgs; [
go_1_24
gopls
gotools
go-tools
golangci-lint
postgresql
sqlite
];
packages.x86_64-linux.default = self.packages.x86_64-linux.hello; shellHook = ''
echo "LabMCP development shell"
echo "Go version: $(go version)"
'';
};
});
}; nixosModules = {
nixos-options-mcp = { pkgs, ... }: {
imports = [ ./nix/module.nix ];
services.nixos-options-mcp.package = lib.mkDefault self.packages.${pkgs.system}.nixos-options;
};
default = self.nixosModules.nixos-options-mcp;
};
};
} }

27
go.mod
View File

@@ -1,3 +1,28 @@
module git.t-juice.club/torjus/labmcp module git.t-juice.club/torjus/labmcp
go 1.25.5 go 1.24
require (
github.com/lib/pq v1.10.9
github.com/urfave/cli/v2 v2.27.5
modernc.org/sqlite v1.34.4
)
require (
github.com/cpuguy83/go-md2man/v2 v2.0.5 // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/hashicorp/golang-lru/v2 v2.0.7 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/ncruces/go-strftime v0.1.9 // indirect
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
github.com/russross/blackfriday/v2 v2.1.0 // indirect
github.com/xrash/smetrics v0.0.0-20240521201337-686a1a2994c1 // indirect
golang.org/x/sys v0.22.0 // indirect
modernc.org/gc/v3 v3.0.0-20240107210532-573471604cb6 // indirect
modernc.org/libc v1.55.3 // indirect
modernc.org/mathutil v1.6.0 // indirect
modernc.org/memory v1.8.0 // indirect
modernc.org/strutil v1.2.0 // indirect
modernc.org/token v1.1.0 // indirect
)

59
go.sum Normal file
View File

@@ -0,0 +1,59 @@
github.com/cpuguy83/go-md2man/v2 v2.0.5 h1:ZtcqGrnekaHpVLArFSe4HK5DoKx1T0rq2DwVB0alcyc=
github.com/cpuguy83/go-md2man/v2 v2.0.5/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
github.com/google/pprof v0.0.0-20240409012703-83162a5b38cd h1:gbpYu9NMq8jhDVbvlGkMFWCjLFlqqEZjEmObmhUy6Vo=
github.com/google/pprof v0.0.0-20240409012703-83162a5b38cd/go.mod h1:kf6iHlnVGwgKolg33glAes7Yg/8iWP8ukqeldJSO7jw=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
github.com/lib/pq v1.10.9 h1:YXG7RB+JIjhP29X+OtkiDnYaXQwpS4JEWq7dtCCRUEw=
github.com/lib/pq v1.10.9/go.mod h1:AlVN5x4E4T544tWzH6hKfbfQvm3HdbOxrmggDNAPY9o=
github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
github.com/ncruces/go-strftime v0.1.9 h1:bY0MQC28UADQmHmaF5dgpLmImcShSi2kHU9XLdhx/f4=
github.com/ncruces/go-strftime v0.1.9/go.mod h1:Fwc5htZGVVkseilnfgOVb9mKy6w1naJmn9CehxcKcls=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE=
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
github.com/russross/blackfriday/v2 v2.1.0 h1:JIOH55/0cWyOuilr9/qlrm0BSXldqnqwMsf35Ld67mk=
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
github.com/urfave/cli/v2 v2.27.5 h1:WoHEJLdsXr6dDWoJgMq/CboDmyY/8HMMH1fTECbih+w=
github.com/urfave/cli/v2 v2.27.5/go.mod h1:3Sevf16NykTbInEnD0yKkjDAeZDS0A6bzhBH5hrMvTQ=
github.com/xrash/smetrics v0.0.0-20240521201337-686a1a2994c1 h1:gEOO8jv9F4OT7lGCjxCBTO/36wtF6j2nSip77qHd4x4=
github.com/xrash/smetrics v0.0.0-20240521201337-686a1a2994c1/go.mod h1:Ohn+xnUBiLI6FVj/9LpzZWtj1/D6lUovWYBkxHVV3aM=
golang.org/x/mod v0.16.0 h1:QX4fJ0Rr5cPQCF7O9lh9Se4pmwfwskqZfq5moyldzic=
golang.org/x/mod v0.16.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.22.0 h1:RI27ohtqKCnwULzJLqkv897zojh5/DwS/ENaMzUOaWI=
golang.org/x/sys v0.22.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/tools v0.19.0 h1:tfGCXNR1OsFG+sVdLAitlpjAvD/I6dHDKnYrpEZUHkw=
golang.org/x/tools v0.19.0/go.mod h1:qoJWxmGSIBmAeriMx19ogtrEPrGtDbPK634QFIcLAhc=
modernc.org/cc/v4 v4.21.4 h1:3Be/Rdo1fpr8GrQ7IVw9OHtplU4gWbb+wNgeoBMmGLQ=
modernc.org/cc/v4 v4.21.4/go.mod h1:HM7VJTZbUCR3rV8EYBi9wxnJ0ZBRiGE5OeGXNA0IsLQ=
modernc.org/ccgo/v4 v4.19.2 h1:lwQZgvboKD0jBwdaeVCTouxhxAyN6iawF3STraAal8Y=
modernc.org/ccgo/v4 v4.19.2/go.mod h1:ysS3mxiMV38XGRTTcgo0DQTeTmAO4oCmJl1nX9VFI3s=
modernc.org/fileutil v1.3.0 h1:gQ5SIzK3H9kdfai/5x41oQiKValumqNTDXMvKo62HvE=
modernc.org/fileutil v1.3.0/go.mod h1:XatxS8fZi3pS8/hKG2GH/ArUogfxjpEKs3Ku3aK4JyQ=
modernc.org/gc/v2 v2.4.1 h1:9cNzOqPyMJBvrUipmynX0ZohMhcxPtMccYgGOJdOiBw=
modernc.org/gc/v2 v2.4.1/go.mod h1:wzN5dK1AzVGoH6XOzc3YZ+ey/jPgYHLuVckd62P0GYU=
modernc.org/gc/v3 v3.0.0-20240107210532-573471604cb6 h1:5D53IMaUuA5InSeMu9eJtlQXS2NxAhyWQvkKEgXZhHI=
modernc.org/gc/v3 v3.0.0-20240107210532-573471604cb6/go.mod h1:Qz0X07sNOR1jWYCrJMEnbW/X55x206Q7Vt4mz6/wHp4=
modernc.org/libc v1.55.3 h1:AzcW1mhlPNrRtjS5sS+eW2ISCgSOLLNyFzRh/V3Qj/U=
modernc.org/libc v1.55.3/go.mod h1:qFXepLhz+JjFThQ4kzwzOjA/y/artDeg+pcYnY+Q83w=
modernc.org/mathutil v1.6.0 h1:fRe9+AmYlaej+64JsEEhoWuAYBkOtQiMEU7n/XgfYi4=
modernc.org/mathutil v1.6.0/go.mod h1:Ui5Q9q1TR2gFm0AQRqQUaBWFLAhQpCwNcuhBOSedWPo=
modernc.org/memory v1.8.0 h1:IqGTL6eFMaDZZhEWwcREgeMXYwmW83LYW8cROZYkg+E=
modernc.org/memory v1.8.0/go.mod h1:XPZ936zp5OMKGWPqbD3JShgd/ZoQ7899TUuQqxY+peU=
modernc.org/opt v0.1.3 h1:3XOZf2yznlhC+ibLltsDGzABUGVx8J6pnFMS3E4dcq4=
modernc.org/opt v0.1.3/go.mod h1:WdSiB5evDcignE70guQKxYUl14mgWtbClRi5wmkkTX0=
modernc.org/sortutil v1.2.0 h1:jQiD3PfS2REGJNzNCMMaLSp/wdMNieTbKX920Cqdgqc=
modernc.org/sortutil v1.2.0/go.mod h1:TKU2s7kJMf1AE84OoiGppNHJwvB753OYfNl2WRb++Ss=
modernc.org/sqlite v1.34.4 h1:sjdARozcL5KJBvYQvLlZEmctRgW9xqIZc2ncN7PU0P8=
modernc.org/sqlite v1.34.4/go.mod h1:3QQFCG2SEMtc2nv+Wq4cQCH7Hjcg+p/RMlS1XK+zwbk=
modernc.org/strutil v1.2.0 h1:agBi9dp1I+eOnxXeiZawM8F4LawKv4NzGWSaLfyeNZA=
modernc.org/strutil v1.2.0/go.mod h1:/mdcBmfOibveCTBxUl5B5l6W+TTH1FXPLHZE6bTosX0=
modernc.org/token v1.1.0 h1:Xl7Ap9dKaEs5kLoOQeQmPWevfnk/DM5qcLcYlA8ys6Y=
modernc.org/token v1.1.0/go.mod h1:UGzOrNV1mAFSEB63lOFHIpNRUVMvYTc6yu1SMY/XTDM=

View File

@@ -0,0 +1,246 @@
package database
import (
"context"
"fmt"
"testing"
"time"
)
func BenchmarkCreateOptions(b *testing.B) {
store, err := NewSQLiteStore(":memory:")
if err != nil {
b.Fatalf("Failed to create store: %v", err)
}
defer store.Close()
ctx := context.Background()
if err := store.Initialize(ctx); err != nil {
b.Fatalf("Failed to initialize: %v", err)
}
rev := &Revision{GitHash: "bench123", ChannelName: "bench"}
if err := store.CreateRevision(ctx, rev); err != nil {
b.Fatalf("Failed to create revision: %v", err)
}
b.ResetTimer()
for i := 0; i < b.N; i++ {
opt := &Option{
RevisionID: rev.ID,
Name: fmt.Sprintf("services.test%d.enable", i),
ParentPath: fmt.Sprintf("services.test%d", i),
Type: "boolean",
DefaultValue: "false",
Description: "Test option",
}
if err := store.CreateOption(ctx, opt); err != nil {
b.Fatalf("Failed to create option: %v", err)
}
}
}
func BenchmarkCreateOptionsBatch(b *testing.B) {
benchmarkBatch(b, 100)
}
func BenchmarkCreateOptionsBatch1000(b *testing.B) {
benchmarkBatch(b, 1000)
}
func benchmarkBatch(b *testing.B, batchSize int) {
store, err := NewSQLiteStore(":memory:")
if err != nil {
b.Fatalf("Failed to create store: %v", err)
}
defer store.Close()
ctx := context.Background()
if err := store.Initialize(ctx); err != nil {
b.Fatalf("Failed to initialize: %v", err)
}
rev := &Revision{GitHash: "batchbench", ChannelName: "bench"}
if err := store.CreateRevision(ctx, rev); err != nil {
b.Fatalf("Failed to create revision: %v", err)
}
opts := make([]*Option, batchSize)
for i := 0; i < batchSize; i++ {
opts[i] = &Option{
RevisionID: rev.ID,
Name: fmt.Sprintf("services.batch%d.enable", i),
ParentPath: fmt.Sprintf("services.batch%d", i),
Type: "boolean",
DefaultValue: "false",
Description: "Batch test option",
}
}
b.ResetTimer()
for i := 0; i < b.N; i++ {
// Reset IDs for next iteration
for _, opt := range opts {
opt.ID = 0
}
if err := store.CreateOptionsBatch(ctx, opts); err != nil {
b.Fatalf("Failed to create batch: %v", err)
}
// Clean up for next iteration
store.DeleteRevision(ctx, rev.ID)
rev = &Revision{GitHash: fmt.Sprintf("batchbench%d", i), ChannelName: "bench"}
store.CreateRevision(ctx, rev)
for _, opt := range opts {
opt.RevisionID = rev.ID
}
}
}
func BenchmarkSearchOptions(b *testing.B) {
store, err := NewSQLiteStore(":memory:")
if err != nil {
b.Fatalf("Failed to create store: %v", err)
}
defer store.Close()
ctx := context.Background()
if err := store.Initialize(ctx); err != nil {
b.Fatalf("Failed to initialize: %v", err)
}
rev := &Revision{GitHash: "searchbench", ChannelName: "bench"}
if err := store.CreateRevision(ctx, rev); err != nil {
b.Fatalf("Failed to create revision: %v", err)
}
// Create 1000 options to search through
opts := make([]*Option, 1000)
for i := 0; i < 1000; i++ {
opts[i] = &Option{
RevisionID: rev.ID,
Name: fmt.Sprintf("services.service%d.enable", i),
ParentPath: fmt.Sprintf("services.service%d", i),
Type: "boolean",
DefaultValue: "false",
Description: fmt.Sprintf("Enable service %d for testing purposes", i),
}
}
if err := store.CreateOptionsBatch(ctx, opts); err != nil {
b.Fatalf("Failed to create options: %v", err)
}
b.ResetTimer()
for i := 0; i < b.N; i++ {
_, err := store.SearchOptions(ctx, rev.ID, "enable service", SearchFilters{Limit: 50})
if err != nil {
b.Fatalf("Search failed: %v", err)
}
}
}
func BenchmarkGetChildren(b *testing.B) {
store, err := NewSQLiteStore(":memory:")
if err != nil {
b.Fatalf("Failed to create store: %v", err)
}
defer store.Close()
ctx := context.Background()
if err := store.Initialize(ctx); err != nil {
b.Fatalf("Failed to initialize: %v", err)
}
rev := &Revision{GitHash: "childrenbench", ChannelName: "bench"}
if err := store.CreateRevision(ctx, rev); err != nil {
b.Fatalf("Failed to create revision: %v", err)
}
// Create parent and 100 children
opts := make([]*Option, 101)
opts[0] = &Option{
RevisionID: rev.ID,
Name: "services",
ParentPath: "",
Type: "attrsOf",
}
for i := 1; i <= 100; i++ {
opts[i] = &Option{
RevisionID: rev.ID,
Name: fmt.Sprintf("services.service%d", i),
ParentPath: "services",
Type: "submodule",
}
}
if err := store.CreateOptionsBatch(ctx, opts); err != nil {
b.Fatalf("Failed to create options: %v", err)
}
b.ResetTimer()
for i := 0; i < b.N; i++ {
_, err := store.GetChildren(ctx, rev.ID, "services")
if err != nil {
b.Fatalf("GetChildren failed: %v", err)
}
}
}
func BenchmarkSchemaInitialize(b *testing.B) {
for i := 0; i < b.N; i++ {
store, err := NewSQLiteStore(":memory:")
if err != nil {
b.Fatalf("Failed to create store: %v", err)
}
ctx := context.Background()
if err := store.Initialize(ctx); err != nil {
b.Fatalf("Failed to initialize: %v", err)
}
store.Close()
}
}
// BenchmarkRevisionCRUD benchmarks the full CRUD cycle for revisions.
func BenchmarkRevisionCRUD(b *testing.B) {
store, err := NewSQLiteStore(":memory:")
if err != nil {
b.Fatalf("Failed to create store: %v", err)
}
defer store.Close()
ctx := context.Background()
if err := store.Initialize(ctx); err != nil {
b.Fatalf("Failed to initialize: %v", err)
}
b.ResetTimer()
for i := 0; i < b.N; i++ {
rev := &Revision{
GitHash: fmt.Sprintf("crud%d", i),
ChannelName: "test",
CommitDate: time.Now(),
}
// Create
if err := store.CreateRevision(ctx, rev); err != nil {
b.Fatalf("Create failed: %v", err)
}
// Read
_, err := store.GetRevision(ctx, rev.GitHash)
if err != nil {
b.Fatalf("Get failed: %v", err)
}
// Update
if err := store.UpdateRevisionOptionCount(ctx, rev.ID, 100); err != nil {
b.Fatalf("Update failed: %v", err)
}
// Delete
if err := store.DeleteRevision(ctx, rev.ID); err != nil {
b.Fatalf("Delete failed: %v", err)
}
}
}

View File

@@ -0,0 +1,529 @@
package database
import (
"context"
"testing"
"time"
)
// runStoreTests runs the test suite against a Store implementation.
func runStoreTests(t *testing.T, newStore func(t *testing.T) Store) {
tests := []struct {
name string
test func(t *testing.T, store Store)
}{
{"Initialize", testInitialize},
{"Revisions", testRevisions},
{"Options", testOptions},
{"OptionsSearch", testOptionsSearch},
{"OptionChildren", testOptionChildren},
{"Declarations", testDeclarations},
{"Files", testFiles},
{"SchemaVersion", testSchemaVersion},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
store := newStore(t)
defer store.Close()
tt.test(t, store)
})
}
}
func testInitialize(t *testing.T, store Store) {
ctx := context.Background()
// Initialize should work
if err := store.Initialize(ctx); err != nil {
t.Fatalf("Initialize failed: %v", err)
}
// Initialize again should be idempotent
if err := store.Initialize(ctx); err != nil {
t.Fatalf("Second Initialize failed: %v", err)
}
}
func testRevisions(t *testing.T, store Store) {
ctx := context.Background()
if err := store.Initialize(ctx); err != nil {
t.Fatalf("Initialize failed: %v", err)
}
// Create a revision
rev := &Revision{
GitHash: "abc123def456",
ChannelName: "nixos-unstable",
CommitDate: time.Now().UTC().Truncate(time.Second),
OptionCount: 0,
}
if err := store.CreateRevision(ctx, rev); err != nil {
t.Fatalf("CreateRevision failed: %v", err)
}
if rev.ID == 0 {
t.Error("Expected revision ID to be set")
}
// Get by git hash
got, err := store.GetRevision(ctx, "abc123def456")
if err != nil {
t.Fatalf("GetRevision failed: %v", err)
}
if got == nil {
t.Fatal("Expected revision, got nil")
}
if got.GitHash != rev.GitHash {
t.Errorf("GitHash = %q, want %q", got.GitHash, rev.GitHash)
}
if got.ChannelName != rev.ChannelName {
t.Errorf("ChannelName = %q, want %q", got.ChannelName, rev.ChannelName)
}
// Get by channel name
got, err = store.GetRevisionByChannel(ctx, "nixos-unstable")
if err != nil {
t.Fatalf("GetRevisionByChannel failed: %v", err)
}
if got == nil {
t.Fatal("Expected revision, got nil")
}
if got.ID != rev.ID {
t.Errorf("ID = %d, want %d", got.ID, rev.ID)
}
// Get non-existent
got, err = store.GetRevision(ctx, "nonexistent")
if err != nil {
t.Fatalf("GetRevision for nonexistent failed: %v", err)
}
if got != nil {
t.Error("Expected nil for nonexistent revision")
}
// List revisions
revs, err := store.ListRevisions(ctx)
if err != nil {
t.Fatalf("ListRevisions failed: %v", err)
}
if len(revs) != 1 {
t.Errorf("Expected 1 revision, got %d", len(revs))
}
// Update option count
if err := store.UpdateRevisionOptionCount(ctx, rev.ID, 100); err != nil {
t.Fatalf("UpdateRevisionOptionCount failed: %v", err)
}
got, _ = store.GetRevision(ctx, "abc123def456")
if got.OptionCount != 100 {
t.Errorf("OptionCount = %d, want 100", got.OptionCount)
}
// Delete revision
if err := store.DeleteRevision(ctx, rev.ID); err != nil {
t.Fatalf("DeleteRevision failed: %v", err)
}
revs, _ = store.ListRevisions(ctx)
if len(revs) != 0 {
t.Errorf("Expected 0 revisions after delete, got %d", len(revs))
}
}
func testOptions(t *testing.T, store Store) {
ctx := context.Background()
if err := store.Initialize(ctx); err != nil {
t.Fatalf("Initialize failed: %v", err)
}
// Create a revision first
rev := &Revision{GitHash: "test123", ChannelName: "test"}
if err := store.CreateRevision(ctx, rev); err != nil {
t.Fatalf("CreateRevision failed: %v", err)
}
// Create an option
opt := &Option{
RevisionID: rev.ID,
Name: "services.nginx.enable",
ParentPath: "services.nginx",
Type: "boolean",
DefaultValue: "false",
Description: "Whether to enable nginx.",
ReadOnly: false,
}
if err := store.CreateOption(ctx, opt); err != nil {
t.Fatalf("CreateOption failed: %v", err)
}
if opt.ID == 0 {
t.Error("Expected option ID to be set")
}
// Get option
got, err := store.GetOption(ctx, rev.ID, "services.nginx.enable")
if err != nil {
t.Fatalf("GetOption failed: %v", err)
}
if got == nil {
t.Fatal("Expected option, got nil")
}
if got.Name != opt.Name {
t.Errorf("Name = %q, want %q", got.Name, opt.Name)
}
if got.Type != opt.Type {
t.Errorf("Type = %q, want %q", got.Type, opt.Type)
}
if got.Description != opt.Description {
t.Errorf("Description = %q, want %q", got.Description, opt.Description)
}
// Get non-existent option
got, err = store.GetOption(ctx, rev.ID, "nonexistent")
if err != nil {
t.Fatalf("GetOption for nonexistent failed: %v", err)
}
if got != nil {
t.Error("Expected nil for nonexistent option")
}
// Batch create options
opts := []*Option{
{RevisionID: rev.ID, Name: "services.caddy.enable", ParentPath: "services.caddy", Type: "boolean"},
{RevisionID: rev.ID, Name: "services.caddy.config", ParentPath: "services.caddy", Type: "string"},
}
if err := store.CreateOptionsBatch(ctx, opts); err != nil {
t.Fatalf("CreateOptionsBatch failed: %v", err)
}
for _, o := range opts {
if o.ID == 0 {
t.Errorf("Expected option %q ID to be set", o.Name)
}
}
}
func testOptionsSearch(t *testing.T, store Store) {
ctx := context.Background()
if err := store.Initialize(ctx); err != nil {
t.Fatalf("Initialize failed: %v", err)
}
rev := &Revision{GitHash: "search123", ChannelName: "test"}
if err := store.CreateRevision(ctx, rev); err != nil {
t.Fatalf("CreateRevision failed: %v", err)
}
opts := []*Option{
{RevisionID: rev.ID, Name: "services.nginx.enable", ParentPath: "services.nginx", Type: "boolean", Description: "Enable the nginx web server"},
{RevisionID: rev.ID, Name: "services.nginx.package", ParentPath: "services.nginx", Type: "package", Description: "Nginx package to use"},
{RevisionID: rev.ID, Name: "services.caddy.enable", ParentPath: "services.caddy", Type: "boolean", Description: "Enable the caddy web server"},
{RevisionID: rev.ID, Name: "programs.git.enable", ParentPath: "programs.git", Type: "boolean", Description: "Enable git version control"},
}
if err := store.CreateOptionsBatch(ctx, opts); err != nil {
t.Fatalf("CreateOptionsBatch failed: %v", err)
}
// Search for "nginx"
results, err := store.SearchOptions(ctx, rev.ID, "nginx", SearchFilters{})
if err != nil {
t.Fatalf("SearchOptions failed: %v", err)
}
if len(results) < 1 {
t.Errorf("Expected at least 1 result for 'nginx', got %d", len(results))
}
// Search with namespace filter
results, err = store.SearchOptions(ctx, rev.ID, "enable", SearchFilters{Namespace: "services"})
if err != nil {
t.Fatalf("SearchOptions with namespace failed: %v", err)
}
for _, r := range results {
if r.Name[:8] != "services" {
t.Errorf("Result %q doesn't match namespace filter", r.Name)
}
}
// Search with type filter
results, err = store.SearchOptions(ctx, rev.ID, "nginx", SearchFilters{Type: "boolean"})
if err != nil {
t.Fatalf("SearchOptions with type failed: %v", err)
}
for _, r := range results {
if r.Type != "boolean" {
t.Errorf("Result %q has type %q, expected boolean", r.Name, r.Type)
}
}
// Search with limit
results, err = store.SearchOptions(ctx, rev.ID, "enable", SearchFilters{Limit: 2})
if err != nil {
t.Fatalf("SearchOptions with limit failed: %v", err)
}
if len(results) > 2 {
t.Errorf("Expected at most 2 results, got %d", len(results))
}
// Search with special characters (dots) - should not cause syntax errors
results, err = store.SearchOptions(ctx, rev.ID, "services.nginx", SearchFilters{})
if err != nil {
t.Fatalf("SearchOptions with dots failed: %v", err)
}
if len(results) < 1 {
t.Errorf("Expected at least 1 result for 'services.nginx', got %d", len(results))
}
// Search with other special characters
specialQueries := []string{
"services.nginx.enable",
"nginx:package",
"web-server",
"(nginx)",
}
for _, q := range specialQueries {
_, err = store.SearchOptions(ctx, rev.ID, q, SearchFilters{})
if err != nil {
t.Errorf("SearchOptions with special query %q failed: %v", q, err)
}
}
}
func testOptionChildren(t *testing.T, store Store) {
ctx := context.Background()
if err := store.Initialize(ctx); err != nil {
t.Fatalf("Initialize failed: %v", err)
}
rev := &Revision{GitHash: "children123", ChannelName: "test"}
if err := store.CreateRevision(ctx, rev); err != nil {
t.Fatalf("CreateRevision failed: %v", err)
}
// Create a hierarchy of options
opts := []*Option{
{RevisionID: rev.ID, Name: "services", ParentPath: "", Type: "attrsOf"},
{RevisionID: rev.ID, Name: "services.nginx", ParentPath: "services", Type: "submodule"},
{RevisionID: rev.ID, Name: "services.nginx.enable", ParentPath: "services.nginx", Type: "boolean"},
{RevisionID: rev.ID, Name: "services.nginx.package", ParentPath: "services.nginx", Type: "package"},
{RevisionID: rev.ID, Name: "services.caddy", ParentPath: "services", Type: "submodule"},
}
if err := store.CreateOptionsBatch(ctx, opts); err != nil {
t.Fatalf("CreateOptionsBatch failed: %v", err)
}
// Get children of root
children, err := store.GetChildren(ctx, rev.ID, "")
if err != nil {
t.Fatalf("GetChildren root failed: %v", err)
}
if len(children) != 1 {
t.Errorf("Expected 1 root child, got %d", len(children))
}
if len(children) > 0 && children[0].Name != "services" {
t.Errorf("Expected root child to be 'services', got %q", children[0].Name)
}
// Get children of services
children, err = store.GetChildren(ctx, rev.ID, "services")
if err != nil {
t.Fatalf("GetChildren services failed: %v", err)
}
if len(children) != 2 {
t.Errorf("Expected 2 children of services, got %d", len(children))
}
// Get children of services.nginx
children, err = store.GetChildren(ctx, rev.ID, "services.nginx")
if err != nil {
t.Fatalf("GetChildren services.nginx failed: %v", err)
}
if len(children) != 2 {
t.Errorf("Expected 2 children of services.nginx, got %d", len(children))
}
}
func testDeclarations(t *testing.T, store Store) {
ctx := context.Background()
if err := store.Initialize(ctx); err != nil {
t.Fatalf("Initialize failed: %v", err)
}
rev := &Revision{GitHash: "decl123", ChannelName: "test"}
if err := store.CreateRevision(ctx, rev); err != nil {
t.Fatalf("CreateRevision failed: %v", err)
}
opt := &Option{
RevisionID: rev.ID,
Name: "services.nginx.enable",
ParentPath: "services.nginx",
Type: "boolean",
}
if err := store.CreateOption(ctx, opt); err != nil {
t.Fatalf("CreateOption failed: %v", err)
}
// Create declarations
decl := &Declaration{
OptionID: opt.ID,
FilePath: "nixos/modules/services/web-servers/nginx/default.nix",
Line: 42,
}
if err := store.CreateDeclaration(ctx, decl); err != nil {
t.Fatalf("CreateDeclaration failed: %v", err)
}
if decl.ID == 0 {
t.Error("Expected declaration ID to be set")
}
// Get declarations
decls, err := store.GetDeclarations(ctx, opt.ID)
if err != nil {
t.Fatalf("GetDeclarations failed: %v", err)
}
if len(decls) != 1 {
t.Fatalf("Expected 1 declaration, got %d", len(decls))
}
if decls[0].FilePath != decl.FilePath {
t.Errorf("FilePath = %q, want %q", decls[0].FilePath, decl.FilePath)
}
if decls[0].Line != 42 {
t.Errorf("Line = %d, want 42", decls[0].Line)
}
// Batch create
batch := []*Declaration{
{OptionID: opt.ID, FilePath: "file1.nix", Line: 10},
{OptionID: opt.ID, FilePath: "file2.nix", Line: 20},
}
if err := store.CreateDeclarationsBatch(ctx, batch); err != nil {
t.Fatalf("CreateDeclarationsBatch failed: %v", err)
}
decls, _ = store.GetDeclarations(ctx, opt.ID)
if len(decls) != 3 {
t.Errorf("Expected 3 declarations after batch, got %d", len(decls))
}
}
func testFiles(t *testing.T, store Store) {
ctx := context.Background()
if err := store.Initialize(ctx); err != nil {
t.Fatalf("Initialize failed: %v", err)
}
rev := &Revision{GitHash: "files123", ChannelName: "test"}
if err := store.CreateRevision(ctx, rev); err != nil {
t.Fatalf("CreateRevision failed: %v", err)
}
// Create a file
file := &File{
RevisionID: rev.ID,
FilePath: "nixos/modules/services/web-servers/nginx/default.nix",
Extension: ".nix",
Content: "{ config, lib, pkgs, ... }:\n\n# nginx module\n",
}
if err := store.CreateFile(ctx, file); err != nil {
t.Fatalf("CreateFile failed: %v", err)
}
if file.ID == 0 {
t.Error("Expected file ID to be set")
}
// Get file
got, err := store.GetFile(ctx, rev.ID, "nixos/modules/services/web-servers/nginx/default.nix")
if err != nil {
t.Fatalf("GetFile failed: %v", err)
}
if got == nil {
t.Fatal("Expected file, got nil")
}
if got.Content != file.Content {
t.Errorf("Content mismatch")
}
if got.Extension != ".nix" {
t.Errorf("Extension = %q, want .nix", got.Extension)
}
// Get non-existent file
got, err = store.GetFile(ctx, rev.ID, "nonexistent.nix")
if err != nil {
t.Fatalf("GetFile for nonexistent failed: %v", err)
}
if got != nil {
t.Error("Expected nil for nonexistent file")
}
// Batch create
files := []*File{
{RevisionID: rev.ID, FilePath: "file1.nix", Extension: ".nix", Content: "content1"},
{RevisionID: rev.ID, FilePath: "file2.nix", Extension: ".nix", Content: "content2"},
}
if err := store.CreateFilesBatch(ctx, files); err != nil {
t.Fatalf("CreateFilesBatch failed: %v", err)
}
for _, f := range files {
if f.ID == 0 {
t.Errorf("Expected file %q ID to be set", f.FilePath)
}
}
}
func testSchemaVersion(t *testing.T, store Store) {
ctx := context.Background()
// First initialization
if err := store.Initialize(ctx); err != nil {
t.Fatalf("First Initialize failed: %v", err)
}
// Create some data
rev := &Revision{GitHash: "version123", ChannelName: "test"}
if err := store.CreateRevision(ctx, rev); err != nil {
t.Fatalf("CreateRevision failed: %v", err)
}
// Second initialization should preserve data (same version)
if err := store.Initialize(ctx); err != nil {
t.Fatalf("Second Initialize failed: %v", err)
}
got, err := store.GetRevision(ctx, "version123")
if err != nil {
t.Fatalf("GetRevision after second init failed: %v", err)
}
if got == nil {
t.Error("Data should be preserved after second initialization")
}
}
// TestParentPath tests the ParentPath helper function.
func TestParentPath(t *testing.T) {
tests := []struct {
name string
input string
expect string
}{
{"top level", "services", ""},
{"one level", "services.nginx", "services"},
{"two levels", "services.nginx.enable", "services.nginx"},
{"three levels", "services.nginx.virtualHosts.default", "services.nginx.virtualHosts"},
{"empty", "", ""},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := ParentPath(tt.input)
if got != tt.expect {
t.Errorf("ParentPath(%q) = %q, want %q", tt.input, got, tt.expect)
}
})
}
}

View File

@@ -0,0 +1,88 @@
// Package database provides database abstraction for storing NixOS options.
package database
import (
"context"
"time"
)
// Revision represents an indexed nixpkgs revision.
type Revision struct {
ID int64
GitHash string
ChannelName string
CommitDate time.Time
IndexedAt time.Time
OptionCount int
}
// Option represents a NixOS configuration option.
type Option struct {
ID int64
RevisionID int64
Name string
ParentPath string
Type string
DefaultValue string // JSON text
Example string // JSON text
Description string
ReadOnly bool
}
// Declaration represents a file where an option is declared.
type Declaration struct {
ID int64
OptionID int64
FilePath string
Line int
}
// File represents a cached file from nixpkgs.
type File struct {
ID int64
RevisionID int64
FilePath string
Extension string
Content string
}
// SearchFilters contains optional filters for option search.
type SearchFilters struct {
Type string
Namespace string
HasDefault *bool
Limit int
Offset int
}
// Store defines the interface for database operations.
type Store interface {
// Schema operations
Initialize(ctx context.Context) error
Close() error
// Revision operations
CreateRevision(ctx context.Context, rev *Revision) error
GetRevision(ctx context.Context, gitHash string) (*Revision, error)
GetRevisionByChannel(ctx context.Context, channel string) (*Revision, error)
ListRevisions(ctx context.Context) ([]*Revision, error)
DeleteRevision(ctx context.Context, id int64) error
UpdateRevisionOptionCount(ctx context.Context, id int64, count int) error
// Option operations
CreateOption(ctx context.Context, opt *Option) error
CreateOptionsBatch(ctx context.Context, opts []*Option) error
GetOption(ctx context.Context, revisionID int64, name string) (*Option, error)
GetChildren(ctx context.Context, revisionID int64, parentPath string) ([]*Option, error)
SearchOptions(ctx context.Context, revisionID int64, query string, filters SearchFilters) ([]*Option, error)
// Declaration operations
CreateDeclaration(ctx context.Context, decl *Declaration) error
CreateDeclarationsBatch(ctx context.Context, decls []*Declaration) error
GetDeclarations(ctx context.Context, optionID int64) ([]*Declaration, error)
// File operations
CreateFile(ctx context.Context, file *File) error
CreateFilesBatch(ctx context.Context, files []*File) error
GetFile(ctx context.Context, revisionID int64, path string) (*File, error)
}

View File

@@ -0,0 +1,488 @@
package database
import (
"context"
"database/sql"
"fmt"
"strings"
_ "github.com/lib/pq"
)
// PostgresStore implements Store using PostgreSQL.
type PostgresStore struct {
db *sql.DB
}
// NewPostgresStore creates a new PostgreSQL store.
func NewPostgresStore(connStr string) (*PostgresStore, error) {
db, err := sql.Open("postgres", connStr)
if err != nil {
return nil, fmt.Errorf("failed to open database: %w", err)
}
if err := db.Ping(); err != nil {
db.Close()
return nil, fmt.Errorf("failed to ping database: %w", err)
}
return &PostgresStore{db: db}, nil
}
// Initialize creates or migrates the database schema.
func (s *PostgresStore) Initialize(ctx context.Context) error {
// Check current schema version
var version int
err := s.db.QueryRowContext(ctx,
"SELECT version FROM schema_info LIMIT 1").Scan(&version)
needsRecreate := err != nil || version != SchemaVersion
if needsRecreate {
// Drop all tables in correct order (respecting foreign keys)
dropStmts := []string{
DropDeclarations,
DropOptions,
DropFiles,
DropRevisions,
DropSchemaInfo,
}
for _, stmt := range dropStmts {
if _, err := s.db.ExecContext(ctx, stmt); err != nil {
return fmt.Errorf("failed to drop table: %w", err)
}
}
}
// Create tables
createStmts := []string{
SchemaInfoTable,
// PostgreSQL uses SERIAL for auto-increment
`CREATE TABLE IF NOT EXISTS revisions (
id SERIAL PRIMARY KEY,
git_hash TEXT NOT NULL UNIQUE,
channel_name TEXT,
commit_date TIMESTAMP,
indexed_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
option_count INTEGER NOT NULL DEFAULT 0
)`,
`CREATE TABLE IF NOT EXISTS options (
id SERIAL PRIMARY KEY,
revision_id INTEGER NOT NULL REFERENCES revisions(id) ON DELETE CASCADE,
name TEXT NOT NULL,
parent_path TEXT NOT NULL,
type TEXT,
default_value TEXT,
example TEXT,
description TEXT,
read_only BOOLEAN NOT NULL DEFAULT FALSE
)`,
`CREATE TABLE IF NOT EXISTS declarations (
id SERIAL PRIMARY KEY,
option_id INTEGER NOT NULL REFERENCES options(id) ON DELETE CASCADE,
file_path TEXT NOT NULL,
line INTEGER
)`,
`CREATE TABLE IF NOT EXISTS files (
id SERIAL PRIMARY KEY,
revision_id INTEGER NOT NULL REFERENCES revisions(id) ON DELETE CASCADE,
file_path TEXT NOT NULL,
extension TEXT,
content TEXT NOT NULL
)`,
IndexOptionsRevisionName,
IndexOptionsRevisionParent,
IndexFilesRevisionPath,
IndexDeclarationsOption,
}
for _, stmt := range createStmts {
if _, err := s.db.ExecContext(ctx, stmt); err != nil {
return fmt.Errorf("failed to create schema: %w", err)
}
}
// Create full-text search index for PostgreSQL
_, err = s.db.ExecContext(ctx, `
CREATE INDEX IF NOT EXISTS idx_options_fts
ON options USING GIN(to_tsvector('english', name || ' ' || COALESCE(description, '')))
`)
if err != nil {
return fmt.Errorf("failed to create FTS index: %w", err)
}
// Set schema version
if needsRecreate {
_, err = s.db.ExecContext(ctx,
"INSERT INTO schema_info (version) VALUES ($1)", SchemaVersion)
if err != nil {
return fmt.Errorf("failed to set schema version: %w", err)
}
}
return nil
}
// Close closes the database connection.
func (s *PostgresStore) Close() error {
return s.db.Close()
}
// CreateRevision creates a new revision record.
func (s *PostgresStore) CreateRevision(ctx context.Context, rev *Revision) error {
err := s.db.QueryRowContext(ctx, `
INSERT INTO revisions (git_hash, channel_name, commit_date, option_count)
VALUES ($1, $2, $3, $4)
RETURNING id, indexed_at`,
rev.GitHash, rev.ChannelName, rev.CommitDate, rev.OptionCount,
).Scan(&rev.ID, &rev.IndexedAt)
if err != nil {
return fmt.Errorf("failed to create revision: %w", err)
}
return nil
}
// GetRevision retrieves a revision by git hash.
func (s *PostgresStore) GetRevision(ctx context.Context, gitHash string) (*Revision, error) {
rev := &Revision{}
err := s.db.QueryRowContext(ctx, `
SELECT id, git_hash, channel_name, commit_date, indexed_at, option_count
FROM revisions WHERE git_hash = $1`, gitHash,
).Scan(&rev.ID, &rev.GitHash, &rev.ChannelName, &rev.CommitDate, &rev.IndexedAt, &rev.OptionCount)
if err == sql.ErrNoRows {
return nil, nil
}
if err != nil {
return nil, fmt.Errorf("failed to get revision: %w", err)
}
return rev, nil
}
// GetRevisionByChannel retrieves a revision by channel name.
func (s *PostgresStore) GetRevisionByChannel(ctx context.Context, channel string) (*Revision, error) {
rev := &Revision{}
err := s.db.QueryRowContext(ctx, `
SELECT id, git_hash, channel_name, commit_date, indexed_at, option_count
FROM revisions WHERE channel_name = $1
ORDER BY indexed_at DESC LIMIT 1`, channel,
).Scan(&rev.ID, &rev.GitHash, &rev.ChannelName, &rev.CommitDate, &rev.IndexedAt, &rev.OptionCount)
if err == sql.ErrNoRows {
return nil, nil
}
if err != nil {
return nil, fmt.Errorf("failed to get revision by channel: %w", err)
}
return rev, nil
}
// ListRevisions returns all indexed revisions.
func (s *PostgresStore) ListRevisions(ctx context.Context) ([]*Revision, error) {
rows, err := s.db.QueryContext(ctx, `
SELECT id, git_hash, channel_name, commit_date, indexed_at, option_count
FROM revisions ORDER BY indexed_at DESC`)
if err != nil {
return nil, fmt.Errorf("failed to list revisions: %w", err)
}
defer rows.Close()
var revisions []*Revision
for rows.Next() {
rev := &Revision{}
if err := rows.Scan(&rev.ID, &rev.GitHash, &rev.ChannelName, &rev.CommitDate, &rev.IndexedAt, &rev.OptionCount); err != nil {
return nil, fmt.Errorf("failed to scan revision: %w", err)
}
revisions = append(revisions, rev)
}
return revisions, rows.Err()
}
// DeleteRevision removes a revision and all associated data.
func (s *PostgresStore) DeleteRevision(ctx context.Context, id int64) error {
_, err := s.db.ExecContext(ctx, "DELETE FROM revisions WHERE id = $1", id)
if err != nil {
return fmt.Errorf("failed to delete revision: %w", err)
}
return nil
}
// UpdateRevisionOptionCount updates the option count for a revision.
func (s *PostgresStore) UpdateRevisionOptionCount(ctx context.Context, id int64, count int) error {
_, err := s.db.ExecContext(ctx,
"UPDATE revisions SET option_count = $1 WHERE id = $2", count, id)
if err != nil {
return fmt.Errorf("failed to update option count: %w", err)
}
return nil
}
// CreateOption creates a new option record.
func (s *PostgresStore) CreateOption(ctx context.Context, opt *Option) error {
err := s.db.QueryRowContext(ctx, `
INSERT INTO options (revision_id, name, parent_path, type, default_value, example, description, read_only)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8)
RETURNING id`,
opt.RevisionID, opt.Name, opt.ParentPath, opt.Type, opt.DefaultValue, opt.Example, opt.Description, opt.ReadOnly,
).Scan(&opt.ID)
if err != nil {
return fmt.Errorf("failed to create option: %w", err)
}
return nil
}
// CreateOptionsBatch creates multiple options in a batch.
func (s *PostgresStore) CreateOptionsBatch(ctx context.Context, opts []*Option) error {
tx, err := s.db.BeginTx(ctx, nil)
if err != nil {
return fmt.Errorf("failed to begin transaction: %w", err)
}
defer tx.Rollback()
stmt, err := tx.PrepareContext(ctx, `
INSERT INTO options (revision_id, name, parent_path, type, default_value, example, description, read_only)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8)
RETURNING id`)
if err != nil {
return fmt.Errorf("failed to prepare statement: %w", err)
}
defer stmt.Close()
for _, opt := range opts {
err := stmt.QueryRowContext(ctx,
opt.RevisionID, opt.Name, opt.ParentPath, opt.Type, opt.DefaultValue, opt.Example, opt.Description, opt.ReadOnly,
).Scan(&opt.ID)
if err != nil {
return fmt.Errorf("failed to insert option %s: %w", opt.Name, err)
}
}
return tx.Commit()
}
// GetOption retrieves an option by revision and name.
func (s *PostgresStore) GetOption(ctx context.Context, revisionID int64, name string) (*Option, error) {
opt := &Option{}
err := s.db.QueryRowContext(ctx, `
SELECT id, revision_id, name, parent_path, type, default_value, example, description, read_only
FROM options WHERE revision_id = $1 AND name = $2`, revisionID, name,
).Scan(&opt.ID, &opt.RevisionID, &opt.Name, &opt.ParentPath, &opt.Type, &opt.DefaultValue, &opt.Example, &opt.Description, &opt.ReadOnly)
if err == sql.ErrNoRows {
return nil, nil
}
if err != nil {
return nil, fmt.Errorf("failed to get option: %w", err)
}
return opt, nil
}
// GetChildren retrieves direct children of an option.
func (s *PostgresStore) GetChildren(ctx context.Context, revisionID int64, parentPath string) ([]*Option, error) {
rows, err := s.db.QueryContext(ctx, `
SELECT id, revision_id, name, parent_path, type, default_value, example, description, read_only
FROM options WHERE revision_id = $1 AND parent_path = $2
ORDER BY name`, revisionID, parentPath)
if err != nil {
return nil, fmt.Errorf("failed to get children: %w", err)
}
defer rows.Close()
var options []*Option
for rows.Next() {
opt := &Option{}
if err := rows.Scan(&opt.ID, &opt.RevisionID, &opt.Name, &opt.ParentPath, &opt.Type, &opt.DefaultValue, &opt.Example, &opt.Description, &opt.ReadOnly); err != nil {
return nil, fmt.Errorf("failed to scan option: %w", err)
}
options = append(options, opt)
}
return options, rows.Err()
}
// SearchOptions searches for options matching a query.
func (s *PostgresStore) SearchOptions(ctx context.Context, revisionID int64, query string, filters SearchFilters) ([]*Option, error) {
var baseQuery string
var args []interface{}
argNum := 1
// If the query looks like an option path (contains dots), prioritize name-based matching.
if strings.Contains(query, ".") {
baseQuery = `
SELECT id, revision_id, name, parent_path, type, default_value, example, description, read_only
FROM options
WHERE revision_id = $1
AND (name = $2 OR name LIKE $3)`
args = []interface{}{revisionID, query, query + ".%"}
argNum = 4
} else {
// For non-path queries, use PostgreSQL full-text search
baseQuery = `
SELECT id, revision_id, name, parent_path, type, default_value, example, description, read_only
FROM options
WHERE revision_id = $1
AND to_tsvector('english', name || ' ' || COALESCE(description, '')) @@ plainto_tsquery('english', $2)`
args = []interface{}{revisionID, query}
argNum = 3
}
if filters.Type != "" {
baseQuery += fmt.Sprintf(" AND type = $%d", argNum)
args = append(args, filters.Type)
argNum++
}
if filters.Namespace != "" {
baseQuery += fmt.Sprintf(" AND name LIKE $%d", argNum)
args = append(args, filters.Namespace+"%")
argNum++
}
if filters.HasDefault != nil {
if *filters.HasDefault {
baseQuery += " AND default_value IS NOT NULL"
} else {
baseQuery += " AND default_value IS NULL"
}
}
baseQuery += " ORDER BY name"
if filters.Limit > 0 {
baseQuery += fmt.Sprintf(" LIMIT %d", filters.Limit)
}
if filters.Offset > 0 {
baseQuery += fmt.Sprintf(" OFFSET %d", filters.Offset)
}
rows, err := s.db.QueryContext(ctx, baseQuery, args...)
if err != nil {
return nil, fmt.Errorf("failed to search options: %w", err)
}
defer rows.Close()
var options []*Option
for rows.Next() {
opt := &Option{}
if err := rows.Scan(&opt.ID, &opt.RevisionID, &opt.Name, &opt.ParentPath, &opt.Type, &opt.DefaultValue, &opt.Example, &opt.Description, &opt.ReadOnly); err != nil {
return nil, fmt.Errorf("failed to scan option: %w", err)
}
options = append(options, opt)
}
return options, rows.Err()
}
// CreateDeclaration creates a new declaration record.
func (s *PostgresStore) CreateDeclaration(ctx context.Context, decl *Declaration) error {
err := s.db.QueryRowContext(ctx, `
INSERT INTO declarations (option_id, file_path, line)
VALUES ($1, $2, $3)
RETURNING id`,
decl.OptionID, decl.FilePath, decl.Line,
).Scan(&decl.ID)
if err != nil {
return fmt.Errorf("failed to create declaration: %w", err)
}
return nil
}
// CreateDeclarationsBatch creates multiple declarations in a batch.
func (s *PostgresStore) CreateDeclarationsBatch(ctx context.Context, decls []*Declaration) error {
tx, err := s.db.BeginTx(ctx, nil)
if err != nil {
return fmt.Errorf("failed to begin transaction: %w", err)
}
defer tx.Rollback()
stmt, err := tx.PrepareContext(ctx, `
INSERT INTO declarations (option_id, file_path, line)
VALUES ($1, $2, $3)
RETURNING id`)
if err != nil {
return fmt.Errorf("failed to prepare statement: %w", err)
}
defer stmt.Close()
for _, decl := range decls {
err := stmt.QueryRowContext(ctx, decl.OptionID, decl.FilePath, decl.Line).Scan(&decl.ID)
if err != nil {
return fmt.Errorf("failed to insert declaration: %w", err)
}
}
return tx.Commit()
}
// GetDeclarations retrieves declarations for an option.
func (s *PostgresStore) GetDeclarations(ctx context.Context, optionID int64) ([]*Declaration, error) {
rows, err := s.db.QueryContext(ctx, `
SELECT id, option_id, file_path, line
FROM declarations WHERE option_id = $1`, optionID)
if err != nil {
return nil, fmt.Errorf("failed to get declarations: %w", err)
}
defer rows.Close()
var decls []*Declaration
for rows.Next() {
decl := &Declaration{}
if err := rows.Scan(&decl.ID, &decl.OptionID, &decl.FilePath, &decl.Line); err != nil {
return nil, fmt.Errorf("failed to scan declaration: %w", err)
}
decls = append(decls, decl)
}
return decls, rows.Err()
}
// CreateFile creates a new file record.
func (s *PostgresStore) CreateFile(ctx context.Context, file *File) error {
err := s.db.QueryRowContext(ctx, `
INSERT INTO files (revision_id, file_path, extension, content)
VALUES ($1, $2, $3, $4)
RETURNING id`,
file.RevisionID, file.FilePath, file.Extension, file.Content,
).Scan(&file.ID)
if err != nil {
return fmt.Errorf("failed to create file: %w", err)
}
return nil
}
// CreateFilesBatch creates multiple files in a batch.
func (s *PostgresStore) CreateFilesBatch(ctx context.Context, files []*File) error {
tx, err := s.db.BeginTx(ctx, nil)
if err != nil {
return fmt.Errorf("failed to begin transaction: %w", err)
}
defer tx.Rollback()
stmt, err := tx.PrepareContext(ctx, `
INSERT INTO files (revision_id, file_path, extension, content)
VALUES ($1, $2, $3, $4)
RETURNING id`)
if err != nil {
return fmt.Errorf("failed to prepare statement: %w", err)
}
defer stmt.Close()
for _, file := range files {
err := stmt.QueryRowContext(ctx, file.RevisionID, file.FilePath, file.Extension, file.Content).Scan(&file.ID)
if err != nil {
return fmt.Errorf("failed to insert file: %w", err)
}
}
return tx.Commit()
}
// GetFile retrieves a file by revision and path.
func (s *PostgresStore) GetFile(ctx context.Context, revisionID int64, path string) (*File, error) {
file := &File{}
err := s.db.QueryRowContext(ctx, `
SELECT id, revision_id, file_path, extension, content
FROM files WHERE revision_id = $1 AND file_path = $2`, revisionID, path,
).Scan(&file.ID, &file.RevisionID, &file.FilePath, &file.Extension, &file.Content)
if err == sql.ErrNoRows {
return nil, nil
}
if err != nil {
return nil, fmt.Errorf("failed to get file: %w", err)
}
return file, nil
}

View File

@@ -0,0 +1,21 @@
package database
import (
"os"
"testing"
)
func TestPostgresStore(t *testing.T) {
connStr := os.Getenv("TEST_POSTGRES_CONN")
if connStr == "" {
t.Skip("TEST_POSTGRES_CONN not set, skipping PostgreSQL tests")
}
runStoreTests(t, func(t *testing.T) Store {
store, err := NewPostgresStore(connStr)
if err != nil {
t.Fatalf("Failed to create PostgreSQL store: %v", err)
}
return store
})
}

102
internal/database/schema.go Normal file
View File

@@ -0,0 +1,102 @@
package database
// SchemaVersion is the current database schema version.
// When this changes, the database will be dropped and recreated.
const SchemaVersion = 1
// Common SQL statements shared between implementations.
const (
// SchemaInfoTable creates the schema version tracking table.
SchemaInfoTable = `
CREATE TABLE IF NOT EXISTS schema_info (
version INTEGER NOT NULL
)`
// RevisionsTable creates the revisions table.
RevisionsTable = `
CREATE TABLE IF NOT EXISTS revisions (
id INTEGER PRIMARY KEY,
git_hash TEXT NOT NULL UNIQUE,
channel_name TEXT,
commit_date TIMESTAMP,
indexed_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
option_count INTEGER NOT NULL DEFAULT 0
)`
// OptionsTable creates the options table.
OptionsTable = `
CREATE TABLE IF NOT EXISTS options (
id INTEGER PRIMARY KEY,
revision_id INTEGER NOT NULL REFERENCES revisions(id) ON DELETE CASCADE,
name TEXT NOT NULL,
parent_path TEXT NOT NULL,
type TEXT,
default_value TEXT,
example TEXT,
description TEXT,
read_only BOOLEAN NOT NULL DEFAULT FALSE
)`
// DeclarationsTable creates the declarations table.
DeclarationsTable = `
CREATE TABLE IF NOT EXISTS declarations (
id INTEGER PRIMARY KEY,
option_id INTEGER NOT NULL REFERENCES options(id) ON DELETE CASCADE,
file_path TEXT NOT NULL,
line INTEGER
)`
// FilesTable creates the files table.
FilesTable = `
CREATE TABLE IF NOT EXISTS files (
id INTEGER PRIMARY KEY,
revision_id INTEGER NOT NULL REFERENCES revisions(id) ON DELETE CASCADE,
file_path TEXT NOT NULL,
extension TEXT,
content TEXT NOT NULL
)`
)
// Index creation statements.
const (
// IndexOptionsRevisionName creates an index on options(revision_id, name).
IndexOptionsRevisionName = `
CREATE INDEX IF NOT EXISTS idx_options_revision_name
ON options(revision_id, name)`
// IndexOptionsRevisionParent creates an index on options(revision_id, parent_path).
IndexOptionsRevisionParent = `
CREATE INDEX IF NOT EXISTS idx_options_revision_parent
ON options(revision_id, parent_path)`
// IndexFilesRevisionPath creates an index on files(revision_id, file_path).
IndexFilesRevisionPath = `
CREATE UNIQUE INDEX IF NOT EXISTS idx_files_revision_path
ON files(revision_id, file_path)`
// IndexDeclarationsOption creates an index on declarations(option_id).
IndexDeclarationsOption = `
CREATE INDEX IF NOT EXISTS idx_declarations_option
ON declarations(option_id)`
)
// Drop statements for schema recreation.
const (
DropSchemaInfo = `DROP TABLE IF EXISTS schema_info`
DropDeclarations = `DROP TABLE IF EXISTS declarations`
DropOptions = `DROP TABLE IF EXISTS options`
DropFiles = `DROP TABLE IF EXISTS files`
DropRevisions = `DROP TABLE IF EXISTS revisions`
)
// ParentPath extracts the parent path from an option name.
// For example, "services.nginx.enable" returns "services.nginx".
// Top-level options return an empty string.
func ParentPath(name string) string {
for i := len(name) - 1; i >= 0; i-- {
if name[i] == '.' {
return name[:i]
}
}
return ""
}

535
internal/database/sqlite.go Normal file
View File

@@ -0,0 +1,535 @@
package database
import (
"context"
"database/sql"
"fmt"
"strings"
_ "modernc.org/sqlite"
)
// SQLiteStore implements Store using SQLite.
type SQLiteStore struct {
db *sql.DB
}
// NewSQLiteStore creates a new SQLite store.
func NewSQLiteStore(path string) (*SQLiteStore, error) {
db, err := sql.Open("sqlite", path)
if err != nil {
return nil, fmt.Errorf("failed to open database: %w", err)
}
// Enable foreign keys
if _, err := db.Exec("PRAGMA foreign_keys = ON"); err != nil {
db.Close()
return nil, fmt.Errorf("failed to enable foreign keys: %w", err)
}
return &SQLiteStore{db: db}, nil
}
// Initialize creates or migrates the database schema.
func (s *SQLiteStore) Initialize(ctx context.Context) error {
// Check current schema version
var version int
err := s.db.QueryRowContext(ctx,
"SELECT version FROM schema_info LIMIT 1").Scan(&version)
needsRecreate := err != nil || version != SchemaVersion
if needsRecreate {
// Drop all tables in correct order (respecting foreign keys)
dropStmts := []string{
DropDeclarations,
DropOptions,
DropFiles,
DropRevisions,
DropSchemaInfo,
"DROP TABLE IF EXISTS options_fts",
}
for _, stmt := range dropStmts {
if _, err := s.db.ExecContext(ctx, stmt); err != nil {
return fmt.Errorf("failed to drop table: %w", err)
}
}
}
// Create tables (SQLite uses INTEGER PRIMARY KEY for auto-increment)
createStmts := []string{
SchemaInfoTable,
RevisionsTable,
OptionsTable,
DeclarationsTable,
FilesTable,
IndexOptionsRevisionName,
IndexOptionsRevisionParent,
IndexFilesRevisionPath,
IndexDeclarationsOption,
}
for _, stmt := range createStmts {
if _, err := s.db.ExecContext(ctx, stmt); err != nil {
return fmt.Errorf("failed to create schema: %w", err)
}
}
// Create FTS5 virtual table for SQLite full-text search
_, err = s.db.ExecContext(ctx, `
CREATE VIRTUAL TABLE IF NOT EXISTS options_fts USING fts5(
name,
description,
content='options',
content_rowid='id'
)
`)
if err != nil {
return fmt.Errorf("failed to create FTS table: %w", err)
}
// Create triggers to keep FTS in sync
triggers := []string{
`CREATE TRIGGER IF NOT EXISTS options_ai AFTER INSERT ON options BEGIN
INSERT INTO options_fts(rowid, name, description) VALUES (new.id, new.name, new.description);
END`,
`CREATE TRIGGER IF NOT EXISTS options_ad AFTER DELETE ON options BEGIN
INSERT INTO options_fts(options_fts, rowid, name, description) VALUES('delete', old.id, old.name, old.description);
END`,
`CREATE TRIGGER IF NOT EXISTS options_au AFTER UPDATE ON options BEGIN
INSERT INTO options_fts(options_fts, rowid, name, description) VALUES('delete', old.id, old.name, old.description);
INSERT INTO options_fts(rowid, name, description) VALUES (new.id, new.name, new.description);
END`,
}
for _, trigger := range triggers {
if _, err := s.db.ExecContext(ctx, trigger); err != nil {
return fmt.Errorf("failed to create trigger: %w", err)
}
}
// Set schema version
if needsRecreate {
_, err = s.db.ExecContext(ctx,
"INSERT INTO schema_info (version) VALUES (?)", SchemaVersion)
if err != nil {
return fmt.Errorf("failed to set schema version: %w", err)
}
}
return nil
}
// Close closes the database connection.
func (s *SQLiteStore) Close() error {
return s.db.Close()
}
// CreateRevision creates a new revision record.
func (s *SQLiteStore) CreateRevision(ctx context.Context, rev *Revision) error {
result, err := s.db.ExecContext(ctx, `
INSERT INTO revisions (git_hash, channel_name, commit_date, option_count)
VALUES (?, ?, ?, ?)`,
rev.GitHash, rev.ChannelName, rev.CommitDate, rev.OptionCount,
)
if err != nil {
return fmt.Errorf("failed to create revision: %w", err)
}
id, err := result.LastInsertId()
if err != nil {
return fmt.Errorf("failed to get last insert id: %w", err)
}
rev.ID = id
// Fetch the indexed_at timestamp
err = s.db.QueryRowContext(ctx,
"SELECT indexed_at FROM revisions WHERE id = ?", id).Scan(&rev.IndexedAt)
if err != nil {
return fmt.Errorf("failed to get indexed_at: %w", err)
}
return nil
}
// GetRevision retrieves a revision by git hash.
func (s *SQLiteStore) GetRevision(ctx context.Context, gitHash string) (*Revision, error) {
rev := &Revision{}
err := s.db.QueryRowContext(ctx, `
SELECT id, git_hash, channel_name, commit_date, indexed_at, option_count
FROM revisions WHERE git_hash = ?`, gitHash,
).Scan(&rev.ID, &rev.GitHash, &rev.ChannelName, &rev.CommitDate, &rev.IndexedAt, &rev.OptionCount)
if err == sql.ErrNoRows {
return nil, nil
}
if err != nil {
return nil, fmt.Errorf("failed to get revision: %w", err)
}
return rev, nil
}
// GetRevisionByChannel retrieves a revision by channel name.
func (s *SQLiteStore) GetRevisionByChannel(ctx context.Context, channel string) (*Revision, error) {
rev := &Revision{}
err := s.db.QueryRowContext(ctx, `
SELECT id, git_hash, channel_name, commit_date, indexed_at, option_count
FROM revisions WHERE channel_name = ?
ORDER BY indexed_at DESC LIMIT 1`, channel,
).Scan(&rev.ID, &rev.GitHash, &rev.ChannelName, &rev.CommitDate, &rev.IndexedAt, &rev.OptionCount)
if err == sql.ErrNoRows {
return nil, nil
}
if err != nil {
return nil, fmt.Errorf("failed to get revision by channel: %w", err)
}
return rev, nil
}
// ListRevisions returns all indexed revisions.
func (s *SQLiteStore) ListRevisions(ctx context.Context) ([]*Revision, error) {
rows, err := s.db.QueryContext(ctx, `
SELECT id, git_hash, channel_name, commit_date, indexed_at, option_count
FROM revisions ORDER BY indexed_at DESC`)
if err != nil {
return nil, fmt.Errorf("failed to list revisions: %w", err)
}
defer rows.Close()
var revisions []*Revision
for rows.Next() {
rev := &Revision{}
if err := rows.Scan(&rev.ID, &rev.GitHash, &rev.ChannelName, &rev.CommitDate, &rev.IndexedAt, &rev.OptionCount); err != nil {
return nil, fmt.Errorf("failed to scan revision: %w", err)
}
revisions = append(revisions, rev)
}
return revisions, rows.Err()
}
// DeleteRevision removes a revision and all associated data.
func (s *SQLiteStore) DeleteRevision(ctx context.Context, id int64) error {
_, err := s.db.ExecContext(ctx, "DELETE FROM revisions WHERE id = ?", id)
if err != nil {
return fmt.Errorf("failed to delete revision: %w", err)
}
return nil
}
// UpdateRevisionOptionCount updates the option count for a revision.
func (s *SQLiteStore) UpdateRevisionOptionCount(ctx context.Context, id int64, count int) error {
_, err := s.db.ExecContext(ctx,
"UPDATE revisions SET option_count = ? WHERE id = ?", count, id)
if err != nil {
return fmt.Errorf("failed to update option count: %w", err)
}
return nil
}
// CreateOption creates a new option record.
func (s *SQLiteStore) CreateOption(ctx context.Context, opt *Option) error {
result, err := s.db.ExecContext(ctx, `
INSERT INTO options (revision_id, name, parent_path, type, default_value, example, description, read_only)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)`,
opt.RevisionID, opt.Name, opt.ParentPath, opt.Type, opt.DefaultValue, opt.Example, opt.Description, opt.ReadOnly,
)
if err != nil {
return fmt.Errorf("failed to create option: %w", err)
}
id, err := result.LastInsertId()
if err != nil {
return fmt.Errorf("failed to get last insert id: %w", err)
}
opt.ID = id
return nil
}
// CreateOptionsBatch creates multiple options in a batch.
func (s *SQLiteStore) CreateOptionsBatch(ctx context.Context, opts []*Option) error {
tx, err := s.db.BeginTx(ctx, nil)
if err != nil {
return fmt.Errorf("failed to begin transaction: %w", err)
}
defer tx.Rollback()
stmt, err := tx.PrepareContext(ctx, `
INSERT INTO options (revision_id, name, parent_path, type, default_value, example, description, read_only)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)`)
if err != nil {
return fmt.Errorf("failed to prepare statement: %w", err)
}
defer stmt.Close()
for _, opt := range opts {
result, err := stmt.ExecContext(ctx,
opt.RevisionID, opt.Name, opt.ParentPath, opt.Type, opt.DefaultValue, opt.Example, opt.Description, opt.ReadOnly,
)
if err != nil {
return fmt.Errorf("failed to insert option %s: %w", opt.Name, err)
}
id, err := result.LastInsertId()
if err != nil {
return fmt.Errorf("failed to get last insert id: %w", err)
}
opt.ID = id
}
return tx.Commit()
}
// GetOption retrieves an option by revision and name.
func (s *SQLiteStore) GetOption(ctx context.Context, revisionID int64, name string) (*Option, error) {
opt := &Option{}
err := s.db.QueryRowContext(ctx, `
SELECT id, revision_id, name, parent_path, type, default_value, example, description, read_only
FROM options WHERE revision_id = ? AND name = ?`, revisionID, name,
).Scan(&opt.ID, &opt.RevisionID, &opt.Name, &opt.ParentPath, &opt.Type, &opt.DefaultValue, &opt.Example, &opt.Description, &opt.ReadOnly)
if err == sql.ErrNoRows {
return nil, nil
}
if err != nil {
return nil, fmt.Errorf("failed to get option: %w", err)
}
return opt, nil
}
// GetChildren retrieves direct children of an option.
func (s *SQLiteStore) GetChildren(ctx context.Context, revisionID int64, parentPath string) ([]*Option, error) {
rows, err := s.db.QueryContext(ctx, `
SELECT id, revision_id, name, parent_path, type, default_value, example, description, read_only
FROM options WHERE revision_id = ? AND parent_path = ?
ORDER BY name`, revisionID, parentPath)
if err != nil {
return nil, fmt.Errorf("failed to get children: %w", err)
}
defer rows.Close()
var options []*Option
for rows.Next() {
opt := &Option{}
if err := rows.Scan(&opt.ID, &opt.RevisionID, &opt.Name, &opt.ParentPath, &opt.Type, &opt.DefaultValue, &opt.Example, &opt.Description, &opt.ReadOnly); err != nil {
return nil, fmt.Errorf("failed to scan option: %w", err)
}
options = append(options, opt)
}
return options, rows.Err()
}
// SearchOptions searches for options matching a query.
func (s *SQLiteStore) SearchOptions(ctx context.Context, revisionID int64, query string, filters SearchFilters) ([]*Option, error) {
var baseQuery string
var args []interface{}
// If the query looks like an option path (contains dots), prioritize name-based matching.
// This ensures "services.nginx" finds "services.nginx.*" options, not random options
// that happen to mention "nginx" in their description.
if strings.Contains(query, ".") {
// Use LIKE-based search for path queries, with ranking:
// 1. Exact match
// 2. Direct children (query.*)
// 3. All descendants (query.*.*)
baseQuery = `
SELECT id, revision_id, name, parent_path, type, default_value, example, description, read_only
FROM options
WHERE revision_id = ?
AND (name = ? OR name LIKE ?)`
args = []interface{}{revisionID, query, query + ".%"}
} else {
// For non-path queries, use FTS5 for full-text search on name and description
baseQuery = `
SELECT o.id, o.revision_id, o.name, o.parent_path, o.type, o.default_value, o.example, o.description, o.read_only
FROM options o
INNER JOIN options_fts fts ON o.id = fts.rowid
WHERE o.revision_id = ?
AND options_fts MATCH ?`
// Escape the query for FTS5 by wrapping in double quotes for literal matching.
escapedQuery := `"` + strings.ReplaceAll(query, `"`, `""`) + `"`
args = []interface{}{revisionID, escapedQuery}
}
// Use table alias for filters (works for both query types)
tbl := ""
if !strings.Contains(query, ".") {
tbl = "o."
}
if filters.Type != "" {
baseQuery += " AND " + tbl + "type = ?"
args = append(args, filters.Type)
}
if filters.Namespace != "" {
baseQuery += " AND " + tbl + "name LIKE ?"
args = append(args, filters.Namespace+"%")
}
if filters.HasDefault != nil {
if *filters.HasDefault {
baseQuery += " AND " + tbl + "default_value IS NOT NULL"
} else {
baseQuery += " AND " + tbl + "default_value IS NULL"
}
}
baseQuery += " ORDER BY " + tbl + "name"
if filters.Limit > 0 {
baseQuery += fmt.Sprintf(" LIMIT %d", filters.Limit)
}
if filters.Offset > 0 {
baseQuery += fmt.Sprintf(" OFFSET %d", filters.Offset)
}
rows, err := s.db.QueryContext(ctx, baseQuery, args...)
if err != nil {
return nil, fmt.Errorf("failed to search options: %w", err)
}
defer rows.Close()
var options []*Option
for rows.Next() {
opt := &Option{}
if err := rows.Scan(&opt.ID, &opt.RevisionID, &opt.Name, &opt.ParentPath, &opt.Type, &opt.DefaultValue, &opt.Example, &opt.Description, &opt.ReadOnly); err != nil {
return nil, fmt.Errorf("failed to scan option: %w", err)
}
options = append(options, opt)
}
return options, rows.Err()
}
// CreateDeclaration creates a new declaration record.
func (s *SQLiteStore) CreateDeclaration(ctx context.Context, decl *Declaration) error {
result, err := s.db.ExecContext(ctx, `
INSERT INTO declarations (option_id, file_path, line)
VALUES (?, ?, ?)`,
decl.OptionID, decl.FilePath, decl.Line,
)
if err != nil {
return fmt.Errorf("failed to create declaration: %w", err)
}
id, err := result.LastInsertId()
if err != nil {
return fmt.Errorf("failed to get last insert id: %w", err)
}
decl.ID = id
return nil
}
// CreateDeclarationsBatch creates multiple declarations in a batch.
func (s *SQLiteStore) CreateDeclarationsBatch(ctx context.Context, decls []*Declaration) error {
tx, err := s.db.BeginTx(ctx, nil)
if err != nil {
return fmt.Errorf("failed to begin transaction: %w", err)
}
defer tx.Rollback()
stmt, err := tx.PrepareContext(ctx, `
INSERT INTO declarations (option_id, file_path, line)
VALUES (?, ?, ?)`)
if err != nil {
return fmt.Errorf("failed to prepare statement: %w", err)
}
defer stmt.Close()
for _, decl := range decls {
result, err := stmt.ExecContext(ctx, decl.OptionID, decl.FilePath, decl.Line)
if err != nil {
return fmt.Errorf("failed to insert declaration: %w", err)
}
id, err := result.LastInsertId()
if err != nil {
return fmt.Errorf("failed to get last insert id: %w", err)
}
decl.ID = id
}
return tx.Commit()
}
// GetDeclarations retrieves declarations for an option.
func (s *SQLiteStore) GetDeclarations(ctx context.Context, optionID int64) ([]*Declaration, error) {
rows, err := s.db.QueryContext(ctx, `
SELECT id, option_id, file_path, line
FROM declarations WHERE option_id = ?`, optionID)
if err != nil {
return nil, fmt.Errorf("failed to get declarations: %w", err)
}
defer rows.Close()
var decls []*Declaration
for rows.Next() {
decl := &Declaration{}
if err := rows.Scan(&decl.ID, &decl.OptionID, &decl.FilePath, &decl.Line); err != nil {
return nil, fmt.Errorf("failed to scan declaration: %w", err)
}
decls = append(decls, decl)
}
return decls, rows.Err()
}
// CreateFile creates a new file record.
func (s *SQLiteStore) CreateFile(ctx context.Context, file *File) error {
result, err := s.db.ExecContext(ctx, `
INSERT INTO files (revision_id, file_path, extension, content)
VALUES (?, ?, ?, ?)`,
file.RevisionID, file.FilePath, file.Extension, file.Content,
)
if err != nil {
return fmt.Errorf("failed to create file: %w", err)
}
id, err := result.LastInsertId()
if err != nil {
return fmt.Errorf("failed to get last insert id: %w", err)
}
file.ID = id
return nil
}
// CreateFilesBatch creates multiple files in a batch.
func (s *SQLiteStore) CreateFilesBatch(ctx context.Context, files []*File) error {
tx, err := s.db.BeginTx(ctx, nil)
if err != nil {
return fmt.Errorf("failed to begin transaction: %w", err)
}
defer tx.Rollback()
stmt, err := tx.PrepareContext(ctx, `
INSERT INTO files (revision_id, file_path, extension, content)
VALUES (?, ?, ?, ?)`)
if err != nil {
return fmt.Errorf("failed to prepare statement: %w", err)
}
defer stmt.Close()
for _, file := range files {
result, err := stmt.ExecContext(ctx, file.RevisionID, file.FilePath, file.Extension, file.Content)
if err != nil {
return fmt.Errorf("failed to insert file: %w", err)
}
id, err := result.LastInsertId()
if err != nil {
return fmt.Errorf("failed to get last insert id: %w", err)
}
file.ID = id
}
return tx.Commit()
}
// GetFile retrieves a file by revision and path.
func (s *SQLiteStore) GetFile(ctx context.Context, revisionID int64, path string) (*File, error) {
file := &File{}
err := s.db.QueryRowContext(ctx, `
SELECT id, revision_id, file_path, extension, content
FROM files WHERE revision_id = ? AND file_path = ?`, revisionID, path,
).Scan(&file.ID, &file.RevisionID, &file.FilePath, &file.Extension, &file.Content)
if err == sql.ErrNoRows {
return nil, nil
}
if err != nil {
return nil, fmt.Errorf("failed to get file: %w", err)
}
return file, nil
}

View File

@@ -0,0 +1,13 @@
package database
import "testing"
func TestSQLiteStore(t *testing.T) {
runStoreTests(t, func(t *testing.T) Store {
store, err := NewSQLiteStore(":memory:")
if err != nil {
t.Fatalf("Failed to create SQLite store: %v", err)
}
return store
})
}

392
internal/mcp/handlers.go Normal file
View File

@@ -0,0 +1,392 @@
package mcp
import (
"context"
"encoding/json"
"fmt"
"path/filepath"
"strings"
"time"
"git.t-juice.club/torjus/labmcp/internal/database"
"git.t-juice.club/torjus/labmcp/internal/nixos"
)
// RegisterHandlers registers all tool handlers on the server.
func (s *Server) RegisterHandlers(indexer *nixos.Indexer) {
s.tools["search_options"] = s.handleSearchOptions
s.tools["get_option"] = s.handleGetOption
s.tools["get_file"] = s.handleGetFile
s.tools["index_revision"] = s.makeIndexHandler(indexer)
s.tools["list_revisions"] = s.handleListRevisions
s.tools["delete_revision"] = s.handleDeleteRevision
}
// handleSearchOptions handles the search_options tool.
func (s *Server) handleSearchOptions(ctx context.Context, args map[string]interface{}) (CallToolResult, error) {
query, _ := args["query"].(string)
if query == "" {
return ErrorContent(fmt.Errorf("query is required")), nil
}
revision, _ := args["revision"].(string)
rev, err := s.resolveRevision(ctx, revision)
if err != nil {
return ErrorContent(err), nil
}
if rev == nil {
return ErrorContent(fmt.Errorf("no indexed revision available")), nil
}
filters := database.SearchFilters{
Limit: 50,
}
if t, ok := args["type"].(string); ok && t != "" {
filters.Type = t
}
if ns, ok := args["namespace"].(string); ok && ns != "" {
filters.Namespace = ns
}
if limit, ok := args["limit"].(float64); ok && limit > 0 {
filters.Limit = int(limit)
}
options, err := s.store.SearchOptions(ctx, rev.ID, query, filters)
if err != nil {
return ErrorContent(fmt.Errorf("search failed: %w", err)), nil
}
// Format results
var sb strings.Builder
sb.WriteString(fmt.Sprintf("Found %d options matching '%s' in revision %s:\n\n", len(options), query, rev.GitHash[:8]))
for _, opt := range options {
sb.WriteString(fmt.Sprintf("## %s\n", opt.Name))
sb.WriteString(fmt.Sprintf("Type: %s\n", opt.Type))
if opt.Description != "" {
desc := opt.Description
if len(desc) > 200 {
desc = desc[:200] + "..."
}
sb.WriteString(fmt.Sprintf("Description: %s\n", desc))
}
sb.WriteString("\n")
}
return CallToolResult{
Content: []Content{TextContent(sb.String())},
}, nil
}
// handleGetOption handles the get_option tool.
func (s *Server) handleGetOption(ctx context.Context, args map[string]interface{}) (CallToolResult, error) {
name, _ := args["name"].(string)
if name == "" {
return ErrorContent(fmt.Errorf("name is required")), nil
}
revision, _ := args["revision"].(string)
rev, err := s.resolveRevision(ctx, revision)
if err != nil {
return ErrorContent(err), nil
}
if rev == nil {
return ErrorContent(fmt.Errorf("no indexed revision available")), nil
}
option, err := s.store.GetOption(ctx, rev.ID, name)
if err != nil {
return ErrorContent(fmt.Errorf("failed to get option: %w", err)), nil
}
if option == nil {
return ErrorContent(fmt.Errorf("option '%s' not found", name)), nil
}
// Get declarations
declarations, err := s.store.GetDeclarations(ctx, option.ID)
if err != nil {
s.logger.Printf("Failed to get declarations: %v", err)
}
// Format result
var sb strings.Builder
sb.WriteString(fmt.Sprintf("# %s\n\n", option.Name))
sb.WriteString(fmt.Sprintf("**Type:** %s\n", option.Type))
if option.Description != "" {
sb.WriteString(fmt.Sprintf("\n**Description:**\n%s\n", option.Description))
}
if option.DefaultValue != "" && option.DefaultValue != "null" {
sb.WriteString(fmt.Sprintf("\n**Default:** `%s`\n", formatJSON(option.DefaultValue)))
}
if option.Example != "" && option.Example != "null" {
sb.WriteString(fmt.Sprintf("\n**Example:** `%s`\n", formatJSON(option.Example)))
}
if option.ReadOnly {
sb.WriteString("\n**Read-only:** Yes\n")
}
if len(declarations) > 0 {
sb.WriteString("\n**Declared in:**\n")
for _, decl := range declarations {
if decl.Line > 0 {
sb.WriteString(fmt.Sprintf("- %s:%d\n", decl.FilePath, decl.Line))
} else {
sb.WriteString(fmt.Sprintf("- %s\n", decl.FilePath))
}
}
}
// Include children if requested (default: true)
includeChildren := true
if ic, ok := args["include_children"].(bool); ok {
includeChildren = ic
}
if includeChildren {
children, err := s.store.GetChildren(ctx, rev.ID, option.Name)
if err != nil {
s.logger.Printf("Failed to get children: %v", err)
}
if len(children) > 0 {
sb.WriteString("\n**Sub-options:**\n")
for _, child := range children {
// Show just the last part of the name
shortName := child.Name
if strings.HasPrefix(child.Name, option.Name+".") {
shortName = child.Name[len(option.Name)+1:]
}
sb.WriteString(fmt.Sprintf("- `%s` (%s)\n", shortName, child.Type))
}
}
}
return CallToolResult{
Content: []Content{TextContent(sb.String())},
}, nil
}
// handleGetFile handles the get_file tool.
func (s *Server) handleGetFile(ctx context.Context, args map[string]interface{}) (CallToolResult, error) {
path, _ := args["path"].(string)
if path == "" {
return ErrorContent(fmt.Errorf("path is required")), nil
}
// Security: validate path to prevent traversal attacks
// Clean the path and check for dangerous patterns
cleanPath := filepath.Clean(path)
if filepath.IsAbs(cleanPath) {
return ErrorContent(fmt.Errorf("invalid path: absolute paths not allowed")), nil
}
if strings.HasPrefix(cleanPath, "..") {
return ErrorContent(fmt.Errorf("invalid path: directory traversal not allowed")), nil
}
// Use the cleaned path for lookup
path = cleanPath
revision, _ := args["revision"].(string)
rev, err := s.resolveRevision(ctx, revision)
if err != nil {
return ErrorContent(err), nil
}
if rev == nil {
return ErrorContent(fmt.Errorf("no indexed revision available")), nil
}
file, err := s.store.GetFile(ctx, rev.ID, path)
if err != nil {
return ErrorContent(fmt.Errorf("failed to get file: %w", err)), nil
}
if file == nil {
return ErrorContent(fmt.Errorf("file '%s' not found (files may not be indexed for this revision)", path)), nil
}
return CallToolResult{
Content: []Content{TextContent(fmt.Sprintf("```%s\n%s\n```", strings.TrimPrefix(file.Extension, "."), file.Content))},
}, nil
}
// makeIndexHandler creates the index_revision handler with the indexer.
func (s *Server) makeIndexHandler(indexer *nixos.Indexer) ToolHandler {
return func(ctx context.Context, args map[string]interface{}) (CallToolResult, error) {
revision, _ := args["revision"].(string)
if revision == "" {
return ErrorContent(fmt.Errorf("revision is required")), nil
}
result, err := indexer.IndexRevision(ctx, revision)
if err != nil {
return ErrorContent(fmt.Errorf("indexing failed: %w", err)), nil
}
// If already indexed, return early with info
if result.AlreadyIndexed {
var sb strings.Builder
sb.WriteString(fmt.Sprintf("Revision already indexed: %s\n", result.Revision.GitHash))
if result.Revision.ChannelName != "" {
sb.WriteString(fmt.Sprintf("Channel: %s\n", result.Revision.ChannelName))
}
sb.WriteString(fmt.Sprintf("Options: %d\n", result.OptionCount))
sb.WriteString(fmt.Sprintf("Indexed at: %s\n", result.Revision.IndexedAt.Format("2006-01-02 15:04")))
return CallToolResult{
Content: []Content{TextContent(sb.String())},
}, nil
}
// Index files by default
fileCount, err := indexer.IndexFiles(ctx, result.Revision.ID, result.Revision.GitHash)
if err != nil {
s.logger.Printf("Warning: file indexing failed: %v", err)
}
var sb strings.Builder
sb.WriteString(fmt.Sprintf("Indexed revision: %s\n", result.Revision.GitHash))
if result.Revision.ChannelName != "" {
sb.WriteString(fmt.Sprintf("Channel: %s\n", result.Revision.ChannelName))
}
sb.WriteString(fmt.Sprintf("Options: %d\n", result.OptionCount))
sb.WriteString(fmt.Sprintf("Files: %d\n", fileCount))
sb.WriteString(fmt.Sprintf("Duration: %s\n", result.Duration.Round(time.Millisecond)))
return CallToolResult{
Content: []Content{TextContent(sb.String())},
}, nil
}
}
// handleListRevisions handles the list_revisions tool.
func (s *Server) handleListRevisions(ctx context.Context, args map[string]interface{}) (CallToolResult, error) {
revisions, err := s.store.ListRevisions(ctx)
if err != nil {
return ErrorContent(fmt.Errorf("failed to list revisions: %w", err)), nil
}
if len(revisions) == 0 {
return CallToolResult{
Content: []Content{TextContent("No revisions indexed. Use index_revision to index a nixpkgs version.")},
}, nil
}
var sb strings.Builder
sb.WriteString(fmt.Sprintf("Indexed revisions (%d):\n\n", len(revisions)))
for _, rev := range revisions {
sb.WriteString(fmt.Sprintf("- **%s**", rev.GitHash[:12]))
if rev.ChannelName != "" {
sb.WriteString(fmt.Sprintf(" (%s)", rev.ChannelName))
}
sb.WriteString(fmt.Sprintf("\n Options: %d, Indexed: %s\n", rev.OptionCount, rev.IndexedAt.Format("2006-01-02 15:04")))
}
return CallToolResult{
Content: []Content{TextContent(sb.String())},
}, nil
}
// handleDeleteRevision handles the delete_revision tool.
func (s *Server) handleDeleteRevision(ctx context.Context, args map[string]interface{}) (CallToolResult, error) {
revision, _ := args["revision"].(string)
if revision == "" {
return ErrorContent(fmt.Errorf("revision is required")), nil
}
rev, err := s.resolveRevision(ctx, revision)
if err != nil {
return ErrorContent(err), nil
}
if rev == nil {
return ErrorContent(fmt.Errorf("revision '%s' not found", revision)), nil
}
if err := s.store.DeleteRevision(ctx, rev.ID); err != nil {
return ErrorContent(fmt.Errorf("failed to delete revision: %w", err)), nil
}
return CallToolResult{
Content: []Content{TextContent(fmt.Sprintf("Deleted revision %s", rev.GitHash))},
}, nil
}
// resolveRevision resolves a revision string to a Revision object.
func (s *Server) resolveRevision(ctx context.Context, revision string) (*database.Revision, error) {
if revision == "" {
// Try to find a default revision
rev, err := s.store.GetRevisionByChannel(ctx, "nixos-stable")
if err != nil {
return nil, err
}
if rev != nil {
return rev, nil
}
// Fall back to any available revision
revs, err := s.store.ListRevisions(ctx)
if err != nil {
return nil, err
}
if len(revs) > 0 {
return revs[0], nil
}
return nil, nil
}
// Try by git hash first
rev, err := s.store.GetRevision(ctx, revision)
if err != nil {
return nil, err
}
if rev != nil {
return rev, nil
}
// Try by channel name
rev, err = s.store.GetRevisionByChannel(ctx, revision)
if err != nil {
return nil, err
}
return rev, nil
}
// formatJSON formats a JSON string for display, handling compact representation.
func formatJSON(s string) string {
if s == "" || s == "null" {
return s
}
// Try to parse and reformat
var v interface{}
if err := json.Unmarshal([]byte(s), &v); err != nil {
return s
}
// For simple values, return as-is
switch val := v.(type) {
case bool, float64, string:
return s
case []interface{}:
if len(val) <= 3 {
return s
}
case map[string]interface{}:
if len(val) <= 3 {
return s
}
}
// For complex values, try to pretty print (truncated)
b, err := json.MarshalIndent(v, "", " ")
if err != nil {
return s
}
result := string(b)
if len(result) > 500 {
result = result[:500] + "..."
}
return result
}

338
internal/mcp/server.go Normal file
View File

@@ -0,0 +1,338 @@
package mcp
import (
"bufio"
"context"
"encoding/json"
"fmt"
"io"
"log"
"git.t-juice.club/torjus/labmcp/internal/database"
)
// Server is an MCP server that handles JSON-RPC requests over stdio.
type Server struct {
store database.Store
tools map[string]ToolHandler
initialized bool
logger *log.Logger
}
// ToolHandler is a function that handles a tool call.
type ToolHandler func(ctx context.Context, args map[string]interface{}) (CallToolResult, error)
// NewServer creates a new MCP server.
func NewServer(store database.Store, logger *log.Logger) *Server {
if logger == nil {
logger = log.New(io.Discard, "", 0)
}
s := &Server{
store: store,
tools: make(map[string]ToolHandler),
logger: logger,
}
s.registerTools()
return s
}
// registerTools registers all available tools.
func (s *Server) registerTools() {
// Tools will be implemented in handlers.go
}
// Run starts the server, reading from r and writing to w.
func (s *Server) Run(ctx context.Context, r io.Reader, w io.Writer) error {
scanner := bufio.NewScanner(r)
encoder := json.NewEncoder(w)
for scanner.Scan() {
select {
case <-ctx.Done():
return ctx.Err()
default:
}
line := scanner.Bytes()
if len(line) == 0 {
continue
}
var req Request
if err := json.Unmarshal(line, &req); err != nil {
s.logger.Printf("Failed to parse request: %v", err)
resp := Response{
JSONRPC: "2.0",
Error: &Error{
Code: ParseError,
Message: "Parse error",
Data: err.Error(),
},
}
if err := encoder.Encode(resp); err != nil {
return fmt.Errorf("failed to write response: %w", err)
}
continue
}
resp := s.handleRequest(ctx, &req)
if resp != nil {
if err := encoder.Encode(resp); err != nil {
return fmt.Errorf("failed to write response: %w", err)
}
}
}
if err := scanner.Err(); err != nil {
return fmt.Errorf("scanner error: %w", err)
}
return nil
}
// handleRequest processes a single request and returns a response.
func (s *Server) handleRequest(ctx context.Context, req *Request) *Response {
s.logger.Printf("Received request: method=%s id=%v", req.Method, req.ID)
switch req.Method {
case MethodInitialize:
return s.handleInitialize(req)
case MethodInitialized:
// This is a notification, no response needed
s.initialized = true
return nil
case MethodToolsList:
return s.handleToolsList(req)
case MethodToolsCall:
return s.handleToolsCall(ctx, req)
default:
return &Response{
JSONRPC: "2.0",
ID: req.ID,
Error: &Error{
Code: MethodNotFound,
Message: "Method not found",
Data: req.Method,
},
}
}
}
// handleInitialize processes the initialize request.
func (s *Server) handleInitialize(req *Request) *Response {
var params InitializeParams
if req.Params != nil {
if err := json.Unmarshal(req.Params, &params); err != nil {
return &Response{
JSONRPC: "2.0",
ID: req.ID,
Error: &Error{
Code: InvalidParams,
Message: "Invalid params",
Data: err.Error(),
},
}
}
}
s.logger.Printf("Client: %s %s, protocol: %s",
params.ClientInfo.Name, params.ClientInfo.Version, params.ProtocolVersion)
result := InitializeResult{
ProtocolVersion: ProtocolVersion,
Capabilities: Capabilities{
Tools: &ToolsCapability{
ListChanged: false,
},
},
ServerInfo: Implementation{
Name: "nixos-options",
Version: "0.1.0",
},
Instructions: `NixOS Options MCP Server - Search and query NixOS configuration options.
If the current project contains a flake.lock file, you can index the exact nixpkgs revision used by the project:
1. Read the flake.lock file to find the nixpkgs "rev" field
2. Call index_revision with that git hash to index options for that specific version
Example: If flake.lock contains "rev": "abc123...", call index_revision with revision "abc123...".
This ensures option documentation matches the nixpkgs version the project actually uses.`,
}
return &Response{
JSONRPC: "2.0",
ID: req.ID,
Result: result,
}
}
// handleToolsList returns the list of available tools.
func (s *Server) handleToolsList(req *Request) *Response {
tools := s.getToolDefinitions()
return &Response{
JSONRPC: "2.0",
ID: req.ID,
Result: ListToolsResult{Tools: tools},
}
}
// getToolDefinitions returns the tool definitions.
func (s *Server) getToolDefinitions() []Tool {
return []Tool{
{
Name: "search_options",
Description: "Search for NixOS configuration options by name or description",
InputSchema: InputSchema{
Type: "object",
Properties: map[string]Property{
"query": {
Type: "string",
Description: "Search query (matches option names and descriptions)",
},
"revision": {
Type: "string",
Description: "Git hash or channel name (e.g., 'nixos-unstable'). Uses default if not specified.",
},
"type": {
Type: "string",
Description: "Filter by option type (e.g., 'boolean', 'string', 'list')",
},
"namespace": {
Type: "string",
Description: "Filter by namespace prefix (e.g., 'services.nginx')",
},
"limit": {
Type: "integer",
Description: "Maximum number of results (default: 50)",
Default: 50,
},
},
Required: []string{"query"},
},
},
{
Name: "get_option",
Description: "Get full details for a specific NixOS option including its children",
InputSchema: InputSchema{
Type: "object",
Properties: map[string]Property{
"name": {
Type: "string",
Description: "Full option path (e.g., 'services.nginx.enable')",
},
"revision": {
Type: "string",
Description: "Git hash or channel name. Uses default if not specified.",
},
"include_children": {
Type: "boolean",
Description: "Include direct children of this option (default: true)",
Default: true,
},
},
Required: []string{"name"},
},
},
{
Name: "get_file",
Description: "Fetch the contents of a file from nixpkgs",
InputSchema: InputSchema{
Type: "object",
Properties: map[string]Property{
"path": {
Type: "string",
Description: "File path relative to nixpkgs root (e.g., 'nixos/modules/services/web-servers/nginx/default.nix')",
},
"revision": {
Type: "string",
Description: "Git hash or channel name. Uses default if not specified.",
},
},
Required: []string{"path"},
},
},
{
Name: "index_revision",
Description: "Index a nixpkgs revision to make its options searchable",
InputSchema: InputSchema{
Type: "object",
Properties: map[string]Property{
"revision": {
Type: "string",
Description: "Git hash (full or short) or channel name (e.g., 'nixos-unstable', 'nixos-24.05')",
},
},
Required: []string{"revision"},
},
},
{
Name: "list_revisions",
Description: "List all indexed nixpkgs revisions",
InputSchema: InputSchema{
Type: "object",
Properties: map[string]Property{},
},
},
{
Name: "delete_revision",
Description: "Delete an indexed revision and all its data",
InputSchema: InputSchema{
Type: "object",
Properties: map[string]Property{
"revision": {
Type: "string",
Description: "Git hash or channel name of the revision to delete",
},
},
Required: []string{"revision"},
},
},
}
}
// handleToolsCall handles a tool invocation.
func (s *Server) handleToolsCall(ctx context.Context, req *Request) *Response {
var params CallToolParams
if err := json.Unmarshal(req.Params, &params); err != nil {
return &Response{
JSONRPC: "2.0",
ID: req.ID,
Error: &Error{
Code: InvalidParams,
Message: "Invalid params",
Data: err.Error(),
},
}
}
s.logger.Printf("Tool call: %s with args %v", params.Name, params.Arguments)
handler, ok := s.tools[params.Name]
if !ok {
return &Response{
JSONRPC: "2.0",
ID: req.ID,
Result: CallToolResult{
Content: []Content{TextContent(fmt.Sprintf("Unknown tool: %s", params.Name))},
IsError: true,
},
}
}
result, err := handler(ctx, params.Arguments)
if err != nil {
s.logger.Printf("Tool error: %v", err)
return &Response{
JSONRPC: "2.0",
ID: req.ID,
Result: ErrorContent(err),
}
}
return &Response{
JSONRPC: "2.0",
ID: req.ID,
Result: result,
}
}

286
internal/mcp/server_test.go Normal file
View File

@@ -0,0 +1,286 @@
package mcp
import (
"bytes"
"context"
"encoding/json"
"io"
"strings"
"testing"
"git.t-juice.club/torjus/labmcp/internal/database"
"git.t-juice.club/torjus/labmcp/internal/nixos"
)
func TestServerInitialize(t *testing.T) {
store := setupTestStore(t)
server := NewServer(store, nil)
input := `{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"test","version":"1.0"}}}`
resp := runRequest(t, server, input)
if resp.Error != nil {
t.Fatalf("Unexpected error: %v", resp.Error)
}
result, ok := resp.Result.(map[string]interface{})
if !ok {
t.Fatalf("Expected map result, got %T", resp.Result)
}
if result["protocolVersion"] != ProtocolVersion {
t.Errorf("protocolVersion = %v, want %v", result["protocolVersion"], ProtocolVersion)
}
serverInfo := result["serverInfo"].(map[string]interface{})
if serverInfo["name"] != "nixos-options" {
t.Errorf("serverInfo.name = %v, want nixos-options", serverInfo["name"])
}
}
func TestServerToolsList(t *testing.T) {
store := setupTestStore(t)
server := NewServer(store, nil)
input := `{"jsonrpc":"2.0","id":1,"method":"tools/list"}`
resp := runRequest(t, server, input)
if resp.Error != nil {
t.Fatalf("Unexpected error: %v", resp.Error)
}
result, ok := resp.Result.(map[string]interface{})
if !ok {
t.Fatalf("Expected map result, got %T", resp.Result)
}
tools, ok := result["tools"].([]interface{})
if !ok {
t.Fatalf("Expected tools array, got %T", result["tools"])
}
// Should have 6 tools
if len(tools) != 6 {
t.Errorf("Expected 6 tools, got %d", len(tools))
}
// Check tool names
expectedTools := map[string]bool{
"search_options": false,
"get_option": false,
"get_file": false,
"index_revision": false,
"list_revisions": false,
"delete_revision": false,
}
for _, tool := range tools {
toolMap := tool.(map[string]interface{})
name := toolMap["name"].(string)
if _, ok := expectedTools[name]; ok {
expectedTools[name] = true
}
}
for name, found := range expectedTools {
if !found {
t.Errorf("Tool %q not found in tools list", name)
}
}
}
func TestServerMethodNotFound(t *testing.T) {
store := setupTestStore(t)
server := NewServer(store, nil)
input := `{"jsonrpc":"2.0","id":1,"method":"unknown/method"}`
resp := runRequest(t, server, input)
if resp.Error == nil {
t.Fatal("Expected error for unknown method")
}
if resp.Error.Code != MethodNotFound {
t.Errorf("Error code = %d, want %d", resp.Error.Code, MethodNotFound)
}
}
func TestServerParseError(t *testing.T) {
store := setupTestStore(t)
server := NewServer(store, nil)
input := `not valid json`
resp := runRequest(t, server, input)
if resp.Error == nil {
t.Fatal("Expected parse error")
}
if resp.Error.Code != ParseError {
t.Errorf("Error code = %d, want %d", resp.Error.Code, ParseError)
}
}
func TestServerNotification(t *testing.T) {
store := setupTestStore(t)
server := NewServer(store, nil)
// Notification (no response expected)
input := `{"jsonrpc":"2.0","method":"notifications/initialized"}`
var output bytes.Buffer
ctx := context.Background()
err := server.Run(ctx, strings.NewReader(input+"\n"), &output)
if err != nil && err != io.EOF {
t.Fatalf("Run failed: %v", err)
}
// Should not produce any output for notifications
if output.Len() > 0 {
t.Errorf("Expected no output for notification, got: %s", output.String())
}
}
func TestGetFilePathValidation(t *testing.T) {
store := setupTestStore(t)
server := setupTestServer(t, store)
// Create a test revision and file
ctx := context.Background()
rev := &database.Revision{
GitHash: "abc123",
OptionCount: 0,
}
if err := store.CreateRevision(ctx, rev); err != nil {
t.Fatalf("Failed to create revision: %v", err)
}
file := &database.File{
RevisionID: rev.ID,
FilePath: "nixos/modules/test.nix",
Extension: ".nix",
Content: "{ }",
}
if err := store.CreateFile(ctx, file); err != nil {
t.Fatalf("Failed to create file: %v", err)
}
tests := []struct {
name string
path string
wantError bool
errorMsg string
}{
// Valid paths
{"valid relative path", "nixos/modules/test.nix", false, ""},
{"valid simple path", "test.nix", false, ""},
// Path traversal attempts
{"dotdot traversal", "../etc/passwd", true, "directory traversal"},
{"dotdot in middle", "nixos/../../../etc/passwd", true, "directory traversal"},
{"multiple dotdot", "../../etc/passwd", true, "directory traversal"},
// Absolute paths
{"absolute unix path", "/etc/passwd", true, "absolute paths"},
// Cleaned paths that become traversal
{"dot slash dotdot", "./../../etc/passwd", true, "directory traversal"},
// Paths that clean to valid (no error expected, but file won't exist)
{"dotdot at end cleans to valid", "nixos/modules/..", false, ""}, // Cleans to "nixos", which is safe
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
input := `{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"get_file","arguments":{"path":"` + tt.path + `","revision":"abc123"}}}`
resp := runRequest(t, server, input)
result, ok := resp.Result.(map[string]interface{})
if !ok {
t.Fatalf("Expected map result, got %T", resp.Result)
}
isError, _ := result["isError"].(bool)
if tt.wantError {
if !isError {
t.Errorf("Expected error for path %q, got success", tt.path)
} else {
content := result["content"].([]interface{})
text := content[0].(map[string]interface{})["text"].(string)
if tt.errorMsg != "" && !strings.Contains(text, tt.errorMsg) {
t.Errorf("Error message %q doesn't contain %q", text, tt.errorMsg)
}
}
} else {
// For valid paths that don't exist, we expect a "not found" error, not a security error
if isError {
content := result["content"].([]interface{})
text := content[0].(map[string]interface{})["text"].(string)
if strings.Contains(text, "traversal") || strings.Contains(text, "absolute") {
t.Errorf("Got security error for valid path %q: %s", tt.path, text)
}
}
}
})
}
}
// Helper functions
func setupTestStore(t *testing.T) database.Store {
t.Helper()
store, err := database.NewSQLiteStore(":memory:")
if err != nil {
t.Fatalf("Failed to create store: %v", err)
}
if err := store.Initialize(context.Background()); err != nil {
t.Fatalf("Failed to initialize store: %v", err)
}
t.Cleanup(func() {
store.Close()
})
return store
}
func setupTestServer(t *testing.T, store database.Store) *Server {
t.Helper()
server := NewServer(store, nil)
indexer := nixos.NewIndexer(store)
server.RegisterHandlers(indexer)
return server
}
func runRequest(t *testing.T, server *Server, input string) *Response {
t.Helper()
var output bytes.Buffer
ctx := context.Background()
// Run with input terminated by newline
err := server.Run(ctx, strings.NewReader(input+"\n"), &output)
if err != nil && err != io.EOF {
t.Fatalf("Run failed: %v", err)
}
if output.Len() == 0 {
t.Fatal("Expected response, got empty output")
}
var resp Response
if err := json.Unmarshal(output.Bytes(), &resp); err != nil {
t.Fatalf("Failed to parse response: %v\nOutput: %s", err, output.String())
}
return &resp
}

140
internal/mcp/types.go Normal file
View File

@@ -0,0 +1,140 @@
// Package mcp implements the Model Context Protocol (MCP) over JSON-RPC.
package mcp
import "encoding/json"
// JSON-RPC 2.0 types
// Request represents a JSON-RPC request.
type Request struct {
JSONRPC string `json:"jsonrpc"`
ID interface{} `json:"id,omitempty"`
Method string `json:"method"`
Params json.RawMessage `json:"params,omitempty"`
}
// Response represents a JSON-RPC response.
type Response struct {
JSONRPC string `json:"jsonrpc"`
ID interface{} `json:"id,omitempty"`
Result interface{} `json:"result,omitempty"`
Error *Error `json:"error,omitempty"`
}
// Error represents a JSON-RPC error.
type Error struct {
Code int `json:"code"`
Message string `json:"message"`
Data interface{} `json:"data,omitempty"`
}
// Standard JSON-RPC error codes
const (
ParseError = -32700
InvalidRequest = -32600
MethodNotFound = -32601
InvalidParams = -32602
InternalError = -32603
)
// MCP Protocol types
// InitializeParams are sent by the client during initialization.
type InitializeParams struct {
ProtocolVersion string `json:"protocolVersion"`
Capabilities Capabilities `json:"capabilities"`
ClientInfo Implementation `json:"clientInfo"`
}
// InitializeResult is returned after successful initialization.
type InitializeResult struct {
ProtocolVersion string `json:"protocolVersion"`
Capabilities Capabilities `json:"capabilities"`
ServerInfo Implementation `json:"serverInfo"`
Instructions string `json:"instructions,omitempty"`
}
// Capabilities describes client or server capabilities.
type Capabilities struct {
Tools *ToolsCapability `json:"tools,omitempty"`
}
// ToolsCapability describes tool-related capabilities.
type ToolsCapability struct {
ListChanged bool `json:"listChanged,omitempty"`
}
// Implementation describes a client or server implementation.
type Implementation struct {
Name string `json:"name"`
Version string `json:"version"`
}
// Tool describes an MCP tool.
type Tool struct {
Name string `json:"name"`
Description string `json:"description,omitempty"`
InputSchema InputSchema `json:"inputSchema"`
}
// InputSchema describes the JSON Schema for tool inputs.
type InputSchema struct {
Type string `json:"type"`
Properties map[string]Property `json:"properties,omitempty"`
Required []string `json:"required,omitempty"`
}
// Property describes a single property in an input schema.
type Property struct {
Type string `json:"type"`
Description string `json:"description,omitempty"`
Enum []string `json:"enum,omitempty"`
Default any `json:"default,omitempty"`
}
// ListToolsResult is returned by tools/list.
type ListToolsResult struct {
Tools []Tool `json:"tools"`
}
// CallToolParams are sent when calling a tool.
type CallToolParams struct {
Name string `json:"name"`
Arguments map[string]interface{} `json:"arguments,omitempty"`
}
// CallToolResult is returned after calling a tool.
type CallToolResult struct {
Content []Content `json:"content"`
IsError bool `json:"isError,omitempty"`
}
// Content represents a piece of content in a tool result.
type Content struct {
Type string `json:"type"`
Text string `json:"text,omitempty"`
}
// TextContent creates a text content item.
func TextContent(text string) Content {
return Content{Type: "text", Text: text}
}
// ErrorContent creates an error content item.
func ErrorContent(err error) CallToolResult {
return CallToolResult{
Content: []Content{TextContent(err.Error())},
IsError: true,
}
}
// MCP method names
const (
MethodInitialize = "initialize"
MethodInitialized = "notifications/initialized"
MethodToolsList = "tools/list"
MethodToolsCall = "tools/call"
)
// Protocol version
const ProtocolVersion = "2024-11-05"

426
internal/nixos/indexer.go Normal file
View File

@@ -0,0 +1,426 @@
package nixos
import (
"archive/tar"
"compress/gzip"
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"os/exec"
"path/filepath"
"regexp"
"strings"
"time"
"git.t-juice.club/torjus/labmcp/internal/database"
)
// revisionPattern validates revision strings to prevent injection attacks.
// Allows: alphanumeric, hyphens, underscores, dots (for channel names like "nixos-24.11"
// and git hashes). Must be 1-64 characters.
var revisionPattern = regexp.MustCompile(`^[a-zA-Z0-9._-]{1,64}$`)
// Indexer handles indexing of nixpkgs revisions.
type Indexer struct {
store database.Store
httpClient *http.Client
}
// NewIndexer creates a new indexer.
func NewIndexer(store database.Store) *Indexer {
return &Indexer{
store: store,
httpClient: &http.Client{
Timeout: 5 * time.Minute,
},
}
}
// IndexResult contains the results of an indexing operation.
type IndexResult struct {
Revision *database.Revision
OptionCount int
FileCount int
Duration time.Duration
AlreadyIndexed bool // True if revision was already indexed (skipped)
}
// ValidateRevision checks if a revision string is safe to use.
// Returns an error if the revision contains potentially dangerous characters.
func ValidateRevision(revision string) error {
if !revisionPattern.MatchString(revision) {
return fmt.Errorf("invalid revision format: must be 1-64 alphanumeric characters, hyphens, underscores, or dots")
}
return nil
}
// IndexRevision indexes a nixpkgs revision by git hash or channel name.
func (idx *Indexer) IndexRevision(ctx context.Context, revision string) (*IndexResult, error) {
start := time.Now()
// Validate revision to prevent injection attacks
if err := ValidateRevision(revision); err != nil {
return nil, err
}
// Resolve channel names to git refs
ref := resolveRevision(revision)
// Check if already indexed
existing, err := idx.store.GetRevision(ctx, ref)
if err != nil {
return nil, fmt.Errorf("failed to check existing revision: %w", err)
}
if existing != nil {
return &IndexResult{
Revision: existing,
OptionCount: existing.OptionCount,
Duration: time.Since(start),
AlreadyIndexed: true,
}, nil
}
// Build options.json using nix
optionsPath, cleanup, err := idx.buildOptions(ctx, ref)
if err != nil {
return nil, fmt.Errorf("failed to build options: %w", err)
}
defer cleanup()
// Parse options.json
optionsFile, err := os.Open(optionsPath)
if err != nil {
return nil, fmt.Errorf("failed to open options.json: %w", err)
}
defer optionsFile.Close()
options, err := ParseOptions(optionsFile)
if err != nil {
return nil, fmt.Errorf("failed to parse options: %w", err)
}
// Get commit info
commitDate, err := idx.getCommitDate(ctx, ref)
if err != nil {
// Non-fatal, use current time
commitDate = time.Now()
}
// Create revision record
rev := &database.Revision{
GitHash: ref,
ChannelName: getChannelName(revision),
CommitDate: commitDate,
OptionCount: len(options),
}
if err := idx.store.CreateRevision(ctx, rev); err != nil {
return nil, fmt.Errorf("failed to create revision: %w", err)
}
// Store options
if err := idx.storeOptions(ctx, rev.ID, options); err != nil {
// Cleanup on failure
idx.store.DeleteRevision(ctx, rev.ID)
return nil, fmt.Errorf("failed to store options: %w", err)
}
return &IndexResult{
Revision: rev,
OptionCount: len(options),
Duration: time.Since(start),
}, nil
}
// ReindexRevision forces re-indexing of a revision, deleting existing data first.
func (idx *Indexer) ReindexRevision(ctx context.Context, revision string) (*IndexResult, error) {
// Validate revision to prevent injection attacks
if err := ValidateRevision(revision); err != nil {
return nil, err
}
ref := resolveRevision(revision)
// Delete existing revision if present
existing, err := idx.store.GetRevision(ctx, ref)
if err != nil {
return nil, fmt.Errorf("failed to check existing revision: %w", err)
}
if existing != nil {
if err := idx.store.DeleteRevision(ctx, existing.ID); err != nil {
return nil, fmt.Errorf("failed to delete existing revision: %w", err)
}
}
// Now index fresh
return idx.IndexRevision(ctx, revision)
}
// buildOptions builds options.json for a nixpkgs revision.
func (idx *Indexer) buildOptions(ctx context.Context, ref string) (string, func(), error) {
// Create temp directory
tmpDir, err := os.MkdirTemp("", "nixos-options-*")
if err != nil {
return "", nil, fmt.Errorf("failed to create temp dir: %w", err)
}
cleanup := func() {
os.RemoveAll(tmpDir)
}
// Build options.json using nix-build
// This evaluates the NixOS options from the specified nixpkgs revision
nixExpr := fmt.Sprintf(`
let
nixpkgs = builtins.fetchTarball {
url = "https://github.com/NixOS/nixpkgs/archive/%s.tar.gz";
};
pkgs = import nixpkgs { config = {}; };
eval = import (nixpkgs + "/nixos/lib/eval-config.nix") {
modules = [];
system = "x86_64-linux";
};
opts = (pkgs.nixosOptionsDoc { options = eval.options; }).optionsJSON;
in opts
`, ref)
cmd := exec.CommandContext(ctx, "nix-build", "--no-out-link", "-E", nixExpr)
cmd.Dir = tmpDir
output, err := cmd.Output()
if err != nil {
cleanup()
if exitErr, ok := err.(*exec.ExitError); ok {
return "", nil, fmt.Errorf("nix-build failed: %s", string(exitErr.Stderr))
}
return "", nil, fmt.Errorf("nix-build failed: %w", err)
}
// The output is the store path containing share/doc/nixos/options.json
storePath := strings.TrimSpace(string(output))
optionsPath := filepath.Join(storePath, "share", "doc", "nixos", "options.json")
if _, err := os.Stat(optionsPath); err != nil {
cleanup()
return "", nil, fmt.Errorf("options.json not found at %s", optionsPath)
}
return optionsPath, cleanup, nil
}
// storeOptions stores parsed options in the database.
func (idx *Indexer) storeOptions(ctx context.Context, revisionID int64, options map[string]*ParsedOption) error {
// Prepare batch of options
opts := make([]*database.Option, 0, len(options))
declsByName := make(map[string][]*database.Declaration)
for name, opt := range options {
dbOpt := &database.Option{
RevisionID: revisionID,
Name: name,
ParentPath: database.ParentPath(name),
Type: opt.Type,
DefaultValue: opt.Default,
Example: opt.Example,
Description: opt.Description,
ReadOnly: opt.ReadOnly,
}
opts = append(opts, dbOpt)
// Prepare declarations for this option
decls := make([]*database.Declaration, 0, len(opt.Declarations))
for _, path := range opt.Declarations {
decls = append(decls, &database.Declaration{
FilePath: path,
})
}
declsByName[name] = decls
}
// Store options in batches
batchSize := 1000
for i := 0; i < len(opts); i += batchSize {
end := i + batchSize
if end > len(opts) {
end = len(opts)
}
batch := opts[i:end]
if err := idx.store.CreateOptionsBatch(ctx, batch); err != nil {
return fmt.Errorf("failed to store options batch: %w", err)
}
}
// Store declarations
for _, opt := range opts {
decls := declsByName[opt.Name]
for _, decl := range decls {
decl.OptionID = opt.ID
}
if len(decls) > 0 {
if err := idx.store.CreateDeclarationsBatch(ctx, decls); err != nil {
return fmt.Errorf("failed to store declarations for %s: %w", opt.Name, err)
}
}
}
return nil
}
// getCommitDate gets the commit date for a git ref.
func (idx *Indexer) getCommitDate(ctx context.Context, ref string) (time.Time, error) {
// Use GitHub API to get commit info
url := fmt.Sprintf("https://api.github.com/repos/NixOS/nixpkgs/commits/%s", ref)
req, err := http.NewRequestWithContext(ctx, "GET", url, nil)
if err != nil {
return time.Time{}, err
}
req.Header.Set("Accept", "application/vnd.github.v3+json")
resp, err := idx.httpClient.Do(req)
if err != nil {
return time.Time{}, err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return time.Time{}, fmt.Errorf("GitHub API returned %d", resp.StatusCode)
}
var commit struct {
Commit struct {
Committer struct {
Date time.Time `json:"date"`
} `json:"committer"`
} `json:"commit"`
}
if err := json.NewDecoder(resp.Body).Decode(&commit); err != nil {
return time.Time{}, err
}
return commit.Commit.Committer.Date, nil
}
// resolveRevision resolves a channel name or ref to a git ref.
func resolveRevision(revision string) string {
// Check if it's a known channel alias
if ref, ok := ChannelAliases[revision]; ok {
return ref
}
return revision
}
// getChannelName returns the channel name if the revision matches one.
func getChannelName(revision string) string {
if _, ok := ChannelAliases[revision]; ok {
return revision
}
// Check if the revision is a channel ref value
for name, ref := range ChannelAliases {
if ref == revision {
return name
}
}
return ""
}
// IndexFiles indexes files from a nixpkgs tarball.
// This is a separate operation that can be run after IndexRevision.
func (idx *Indexer) IndexFiles(ctx context.Context, revisionID int64, ref string) (int, error) {
// Download nixpkgs tarball
url := fmt.Sprintf("https://github.com/NixOS/nixpkgs/archive/%s.tar.gz", ref)
req, err := http.NewRequestWithContext(ctx, "GET", url, nil)
if err != nil {
return 0, fmt.Errorf("failed to create request: %w", err)
}
resp, err := idx.httpClient.Do(req)
if err != nil {
return 0, fmt.Errorf("failed to download tarball: %w", err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return 0, fmt.Errorf("download failed with status %d", resp.StatusCode)
}
// Extract and index files
gz, err := gzip.NewReader(resp.Body)
if err != nil {
return 0, fmt.Errorf("failed to create gzip reader: %w", err)
}
defer gz.Close()
tr := tar.NewReader(gz)
count := 0
batch := make([]*database.File, 0, 100)
for {
header, err := tr.Next()
if err == io.EOF {
break
}
if err != nil {
return count, fmt.Errorf("tar read error: %w", err)
}
// Skip directories
if header.Typeflag != tar.TypeReg {
continue
}
// Check file extension
ext := filepath.Ext(header.Name)
if !AllowedExtensions[ext] {
continue
}
// Skip very large files (> 1MB)
if header.Size > 1024*1024 {
continue
}
// Remove the top-level directory (nixpkgs-<hash>/)
path := header.Name
if idx := strings.Index(path, "/"); idx >= 0 {
path = path[idx+1:]
}
// Read content
content, err := io.ReadAll(tr)
if err != nil {
continue
}
file := &database.File{
RevisionID: revisionID,
FilePath: path,
Extension: ext,
Content: string(content),
}
batch = append(batch, file)
count++
// Store in batches
if len(batch) >= 100 {
if err := idx.store.CreateFilesBatch(ctx, batch); err != nil {
return count, fmt.Errorf("failed to store files batch: %w", err)
}
batch = batch[:0]
}
}
// Store remaining files
if len(batch) > 0 {
if err := idx.store.CreateFilesBatch(ctx, batch); err != nil {
return count, fmt.Errorf("failed to store final files batch: %w", err)
}
}
return count, nil
}

View File

@@ -0,0 +1,277 @@
package nixos
import (
"context"
"os/exec"
"testing"
"time"
"git.t-juice.club/torjus/labmcp/internal/database"
)
// TestNixpkgsRevision is the revision from flake.lock used for testing.
const TestNixpkgsRevision = "e6eae2ee2110f3d31110d5c222cd395303343b08"
// TestValidateRevision tests the revision validation function.
func TestValidateRevision(t *testing.T) {
tests := []struct {
name string
revision string
wantErr bool
}{
// Valid cases
{"valid git hash", "e6eae2ee2110f3d31110d5c222cd395303343b08", false},
{"valid short hash", "e6eae2e", false},
{"valid channel name", "nixos-unstable", false},
{"valid channel with version", "nixos-24.11", false},
{"valid underscore", "some_branch", false},
{"valid mixed", "release-24.05_beta", false},
// Invalid cases - injection attempts
{"injection semicolon", "foo; rm -rf /", true},
{"injection quotes", `"; builtins.readFile /etc/passwd; "`, true},
{"injection backticks", "foo`whoami`", true},
{"injection dollar", "foo$(whoami)", true},
{"injection newline", "foo\nbar", true},
{"injection space", "foo bar", true},
{"injection slash", "foo/bar", true},
{"injection backslash", "foo\\bar", true},
{"injection pipe", "foo|bar", true},
{"injection ampersand", "foo&bar", true},
{"injection redirect", "foo>bar", true},
{"injection less than", "foo<bar", true},
{"injection curly braces", "foo{bar}", true},
{"injection parens", "foo(bar)", true},
{"injection brackets", "foo[bar]", true},
// Edge cases
{"empty string", "", true},
{"too long", "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", true},
{"just dots", "...", false}, // dots are allowed, path traversal is handled elsewhere
{"single char", "a", false},
{"max length 64", "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", false},
{"65 chars", "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", true},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
err := ValidateRevision(tt.revision)
if (err != nil) != tt.wantErr {
t.Errorf("ValidateRevision(%q) error = %v, wantErr %v", tt.revision, err, tt.wantErr)
}
})
}
}
// BenchmarkIndexRevision benchmarks indexing a full nixpkgs revision.
// This is a slow benchmark that requires nix to be installed.
// Run with: go test -bench=BenchmarkIndexRevision -benchtime=1x -timeout=30m ./internal/nixos/...
func BenchmarkIndexRevision(b *testing.B) {
// Check if nix-build is available
if _, err := exec.LookPath("nix-build"); err != nil {
b.Skip("nix-build not found, skipping indexer benchmark")
}
// Use in-memory SQLite for the benchmark
store, err := database.NewSQLiteStore(":memory:")
if err != nil {
b.Fatalf("Failed to create store: %v", err)
}
defer store.Close()
ctx := context.Background()
if err := store.Initialize(ctx); err != nil {
b.Fatalf("Failed to initialize store: %v", err)
}
indexer := NewIndexer(store)
b.ResetTimer()
for i := 0; i < b.N; i++ {
// Delete any existing revision first (for repeated runs)
if rev, _ := store.GetRevision(ctx, TestNixpkgsRevision); rev != nil {
store.DeleteRevision(ctx, rev.ID)
}
result, err := indexer.IndexRevision(ctx, TestNixpkgsRevision)
if err != nil {
b.Fatalf("IndexRevision failed: %v", err)
}
b.ReportMetric(float64(result.OptionCount), "options")
b.ReportMetric(float64(result.Duration.Milliseconds()), "ms")
}
}
// BenchmarkIndexRevisionWithFiles benchmarks indexing with file content storage.
// This downloads the full nixpkgs tarball and stores allowed file types.
// Run with: go test -bench=BenchmarkIndexRevisionWithFiles -benchtime=1x -timeout=60m ./internal/nixos/...
func BenchmarkIndexRevisionWithFiles(b *testing.B) {
// Check if nix-build is available
if _, err := exec.LookPath("nix-build"); err != nil {
b.Skip("nix-build not found, skipping indexer benchmark")
}
// Use in-memory SQLite for the benchmark
store, err := database.NewSQLiteStore(":memory:")
if err != nil {
b.Fatalf("Failed to create store: %v", err)
}
defer store.Close()
ctx := context.Background()
if err := store.Initialize(ctx); err != nil {
b.Fatalf("Failed to initialize store: %v", err)
}
indexer := NewIndexer(store)
b.ResetTimer()
for i := 0; i < b.N; i++ {
// Delete any existing revision first
if rev, _ := store.GetRevision(ctx, TestNixpkgsRevision); rev != nil {
store.DeleteRevision(ctx, rev.ID)
}
result, err := indexer.IndexRevision(ctx, TestNixpkgsRevision)
if err != nil {
b.Fatalf("IndexRevision failed: %v", err)
}
fileStart := time.Now()
fileCount, err := indexer.IndexFiles(ctx, result.Revision.ID, TestNixpkgsRevision)
if err != nil {
b.Fatalf("IndexFiles failed: %v", err)
}
fileDuration := time.Since(fileStart)
b.ReportMetric(float64(result.OptionCount), "options")
b.ReportMetric(float64(result.Duration.Milliseconds()), "options_ms")
b.ReportMetric(float64(fileCount), "files")
b.ReportMetric(float64(fileDuration.Milliseconds()), "files_ms")
}
}
// BenchmarkIndexFilesOnly benchmarks only the file indexing phase.
// Assumes options are already indexed. Useful for measuring file indexing in isolation.
// Run with: go test -bench=BenchmarkIndexFilesOnly -benchtime=1x -timeout=60m ./internal/nixos/...
func BenchmarkIndexFilesOnly(b *testing.B) {
if _, err := exec.LookPath("nix-build"); err != nil {
b.Skip("nix-build not found, skipping indexer benchmark")
}
store, err := database.NewSQLiteStore(":memory:")
if err != nil {
b.Fatalf("Failed to create store: %v", err)
}
defer store.Close()
ctx := context.Background()
if err := store.Initialize(ctx); err != nil {
b.Fatalf("Failed to initialize store: %v", err)
}
indexer := NewIndexer(store)
// Index options first (outside of benchmark timing)
result, err := indexer.IndexRevision(ctx, TestNixpkgsRevision)
if err != nil {
b.Fatalf("IndexRevision failed: %v", err)
}
b.Logf("Pre-indexed %d options", result.OptionCount)
b.ResetTimer()
for i := 0; i < b.N; i++ {
fileCount, err := indexer.IndexFiles(ctx, result.Revision.ID, TestNixpkgsRevision)
if err != nil {
b.Fatalf("IndexFiles failed: %v", err)
}
b.ReportMetric(float64(fileCount), "files")
}
}
// TestIndexRevision is an integration test for the indexer.
// Run with: go test -run=TestIndexRevision -timeout=30m ./internal/nixos/...
func TestIndexRevision(t *testing.T) {
if testing.Short() {
t.Skip("Skipping integration test in short mode")
}
// Check if nix-build is available
if _, err := exec.LookPath("nix-build"); err != nil {
t.Skip("nix-build not found, skipping indexer test")
}
store, err := database.NewSQLiteStore(":memory:")
if err != nil {
t.Fatalf("Failed to create store: %v", err)
}
defer store.Close()
ctx := context.Background()
if err := store.Initialize(ctx); err != nil {
t.Fatalf("Failed to initialize store: %v", err)
}
indexer := NewIndexer(store)
t.Logf("Indexing nixpkgs revision %s...", TestNixpkgsRevision[:12])
result, err := indexer.IndexRevision(ctx, TestNixpkgsRevision)
if err != nil {
t.Fatalf("IndexRevision failed: %v", err)
}
t.Logf("Indexed %d options in %s", result.OptionCount, result.Duration)
// Verify we got a reasonable number of options (NixOS has thousands)
if result.OptionCount < 1000 {
t.Errorf("Expected at least 1000 options, got %d", result.OptionCount)
}
// Verify revision was stored
rev, err := store.GetRevision(ctx, TestNixpkgsRevision)
if err != nil {
t.Fatalf("GetRevision failed: %v", err)
}
if rev == nil {
t.Fatal("Revision not found after indexing")
}
if rev.OptionCount != result.OptionCount {
t.Errorf("Stored option count %d != result count %d", rev.OptionCount, result.OptionCount)
}
// Test searching
options, err := store.SearchOptions(ctx, rev.ID, "nginx", database.SearchFilters{Limit: 10})
if err != nil {
t.Fatalf("SearchOptions failed: %v", err)
}
if len(options) == 0 {
t.Error("Expected to find nginx options")
}
t.Logf("Found %d nginx options", len(options))
// Test getting a specific option
opt, err := store.GetOption(ctx, rev.ID, "services.nginx.enable")
if err != nil {
t.Fatalf("GetOption failed: %v", err)
}
if opt == nil {
t.Error("services.nginx.enable not found")
} else {
t.Logf("services.nginx.enable: type=%s", opt.Type)
if opt.Type != "boolean" {
t.Errorf("Expected type 'boolean', got %q", opt.Type)
}
}
// Test getting children
children, err := store.GetChildren(ctx, rev.ID, "services.nginx")
if err != nil {
t.Fatalf("GetChildren failed: %v", err)
}
if len(children) == 0 {
t.Error("Expected services.nginx to have children")
}
t.Logf("services.nginx has %d direct children", len(children))
}

126
internal/nixos/parser.go Normal file
View File

@@ -0,0 +1,126 @@
package nixos
import (
"encoding/json"
"fmt"
"io"
)
// ParseOptions parses the options.json file from nixpkgs.
// The options.json structure is a map from option name to option definition.
func ParseOptions(r io.Reader) (map[string]*ParsedOption, error) {
var raw map[string]json.RawMessage
if err := json.NewDecoder(r).Decode(&raw); err != nil {
return nil, fmt.Errorf("failed to decode options.json: %w", err)
}
options := make(map[string]*ParsedOption, len(raw))
for name, data := range raw {
opt, err := parseOption(name, data)
if err != nil {
// Log but don't fail - some options might have unusual formats
continue
}
options[name] = opt
}
return options, nil
}
// ParsedOption represents a parsed NixOS option with all its metadata.
type ParsedOption struct {
Name string
Type string
Description string
Default string // JSON-encoded value
Example string // JSON-encoded value
ReadOnly bool
Declarations []string
}
// optionJSON is the internal structure for parsing options.json entries.
type optionJSON struct {
Declarations []string `json:"declarations"`
Default json.RawMessage `json:"default,omitempty"`
Description interface{} `json:"description"` // Can be string or object
Example json.RawMessage `json:"example,omitempty"`
ReadOnly bool `json:"readOnly"`
Type string `json:"type"`
}
// parseOption parses a single option entry.
func parseOption(name string, data json.RawMessage) (*ParsedOption, error) {
var opt optionJSON
if err := json.Unmarshal(data, &opt); err != nil {
return nil, fmt.Errorf("failed to parse option %s: %w", name, err)
}
// Handle description which can be a string or an object with _type: "mdDoc"
description := extractDescription(opt.Description)
// Convert declarations to relative paths
declarations := make([]string, 0, len(opt.Declarations))
for _, d := range opt.Declarations {
declarations = append(declarations, normalizeDeclarationPath(d))
}
return &ParsedOption{
Name: name,
Type: opt.Type,
Description: description,
Default: string(opt.Default),
Example: string(opt.Example),
ReadOnly: opt.ReadOnly,
Declarations: declarations,
}, nil
}
// extractDescription extracts the description string from various formats.
func extractDescription(desc interface{}) string {
switch v := desc.(type) {
case string:
return v
case map[string]interface{}:
// Handle mdDoc format: {"_type": "mdDoc", "text": "..."}
if text, ok := v["text"].(string); ok {
return text
}
// Try "description" key
if text, ok := v["description"].(string); ok {
return text
}
}
return ""
}
// normalizeDeclarationPath converts a full store path to a relative nixpkgs path.
// Input: "/nix/store/xxx-source/nixos/modules/services/web-servers/nginx/default.nix"
// Output: "nixos/modules/services/web-servers/nginx/default.nix"
func normalizeDeclarationPath(path string) string {
// Look for common prefixes and strip them
markers := []string{
"/nixos/",
"/pkgs/",
"/lib/",
"/maintainers/",
}
for _, marker := range markers {
if idx := findSubstring(path, marker); idx >= 0 {
return path[idx+1:] // +1 to skip the leading /
}
}
// If no marker found, return as-is
return path
}
// findSubstring returns the index of the first occurrence of substr in s, or -1.
func findSubstring(s, substr string) int {
for i := 0; i <= len(s)-len(substr); i++ {
if s[i:i+len(substr)] == substr {
return i
}
}
return -1
}

View File

@@ -0,0 +1,154 @@
package nixos
import (
"strings"
"testing"
)
func TestParseOptions(t *testing.T) {
// Sample options.json content
input := `{
"services.nginx.enable": {
"declarations": ["/nix/store/xxx-source/nixos/modules/services/web-servers/nginx/default.nix"],
"default": false,
"description": "Whether to enable Nginx Web Server.",
"readOnly": false,
"type": "boolean"
},
"services.nginx.package": {
"declarations": ["/nix/store/xxx-source/nixos/modules/services/web-servers/nginx/default.nix"],
"default": {},
"description": {"_type": "mdDoc", "text": "The nginx package to use."},
"readOnly": false,
"type": "package"
},
"services.caddy.enable": {
"declarations": ["/nix/store/xxx-source/nixos/modules/services/web-servers/caddy/default.nix"],
"default": false,
"description": "Enable Caddy web server",
"readOnly": false,
"type": "boolean"
}
}`
options, err := ParseOptions(strings.NewReader(input))
if err != nil {
t.Fatalf("ParseOptions failed: %v", err)
}
if len(options) != 3 {
t.Errorf("Expected 3 options, got %d", len(options))
}
// Check nginx.enable
opt := options["services.nginx.enable"]
if opt == nil {
t.Fatal("Expected services.nginx.enable option")
}
if opt.Type != "boolean" {
t.Errorf("Type = %q, want boolean", opt.Type)
}
if opt.Description != "Whether to enable Nginx Web Server." {
t.Errorf("Description = %q", opt.Description)
}
if opt.Default != "false" {
t.Errorf("Default = %q, want false", opt.Default)
}
if len(opt.Declarations) != 1 {
t.Errorf("Expected 1 declaration, got %d", len(opt.Declarations))
}
// Check declaration path normalization
if !strings.HasPrefix(opt.Declarations[0], "nixos/") {
t.Errorf("Declaration path not normalized: %q", opt.Declarations[0])
}
// Check nginx.package (mdDoc description)
opt = options["services.nginx.package"]
if opt == nil {
t.Fatal("Expected services.nginx.package option")
}
if opt.Description != "The nginx package to use." {
t.Errorf("Description = %q (mdDoc not extracted)", opt.Description)
}
}
func TestNormalizeDeclarationPath(t *testing.T) {
tests := []struct {
input string
expect string
}{
{
"/nix/store/xxx-source/nixos/modules/services/web-servers/nginx/default.nix",
"nixos/modules/services/web-servers/nginx/default.nix",
},
{
"/nix/store/xxx/pkgs/top-level/all-packages.nix",
"pkgs/top-level/all-packages.nix",
},
{
"/nix/store/abc123/lib/types.nix",
"lib/types.nix",
},
{
"relative/path.nix",
"relative/path.nix",
},
}
for _, tt := range tests {
t.Run(tt.input, func(t *testing.T) {
got := normalizeDeclarationPath(tt.input)
if got != tt.expect {
t.Errorf("normalizeDeclarationPath(%q) = %q, want %q", tt.input, got, tt.expect)
}
})
}
}
func TestExtractDescription(t *testing.T) {
tests := []struct {
name string
input interface{}
expect string
}{
{
"string",
"Simple description",
"Simple description",
},
{
"mdDoc",
map[string]interface{}{
"_type": "mdDoc",
"text": "Markdown description",
},
"Markdown description",
},
{
"description key",
map[string]interface{}{
"description": "Nested description",
},
"Nested description",
},
{
"nil",
nil,
"",
},
{
"empty map",
map[string]interface{}{},
"",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := extractDescription(tt.input)
if got != tt.expect {
t.Errorf("extractDescription = %q, want %q", got, tt.expect)
}
})
}
}

45
internal/nixos/types.go Normal file
View File

@@ -0,0 +1,45 @@
// Package nixos contains types and logic specific to NixOS options.
package nixos
// RawOption represents an option as parsed from options.json.
// The structure matches the output of `nix-build '<nixpkgs/nixos/release.nix>' -A options`.
type RawOption struct {
Declarations []string `json:"declarations"`
Default *OptionValue `json:"default,omitempty"`
Description string `json:"description"`
Example *OptionValue `json:"example,omitempty"`
ReadOnly bool `json:"readOnly"`
Type string `json:"type"`
Loc []string `json:"loc,omitempty"`
}
// OptionValue wraps a value that may be a literal or a Nix expression.
type OptionValue struct {
// Text is the raw JSON representation of the value
Text string
}
// OptionsFile represents the top-level structure of options.json.
// It's a map from option name to option definition.
type OptionsFile map[string]RawOption
// AllowedExtensions is the default set of file extensions to index.
var AllowedExtensions = map[string]bool{
".nix": true,
".json": true,
".md": true,
".txt": true,
".toml": true,
".yaml": true,
".yml": true,
}
// ChannelAliases maps friendly channel names to their git branch/ref patterns.
var ChannelAliases = map[string]string{
"nixos-unstable": "nixos-unstable",
"nixos-stable": "nixos-24.11", // Update this as new stable releases come out
"nixos-24.11": "nixos-24.11",
"nixos-24.05": "nixos-24.05",
"nixos-23.11": "nixos-23.11",
"nixos-23.05": "nixos-23.05",
}

192
nix/module.nix Normal file
View File

@@ -0,0 +1,192 @@
{ config, lib, pkgs, ... }:
let
cfg = config.services.nixos-options-mcp;
# Determine database URL based on configuration
# For postgres with connectionStringFile, the URL is set at runtime via script
useConnectionStringFile = cfg.database.type == "postgres" && cfg.database.connectionStringFile != null;
databaseUrl = if cfg.database.type == "sqlite"
then "sqlite://${cfg.dataDir}/${cfg.database.name}"
else if useConnectionStringFile
then "" # Will be set at runtime from file
else cfg.database.connectionString;
in
{
options.services.nixos-options-mcp = {
enable = lib.mkEnableOption "NixOS Options MCP server";
package = lib.mkPackageOption pkgs "nixos-options-mcp" { };
user = lib.mkOption {
type = lib.types.str;
default = "nixos-options-mcp";
description = "User account under which the service runs.";
};
group = lib.mkOption {
type = lib.types.str;
default = "nixos-options-mcp";
description = "Group under which the service runs.";
};
dataDir = lib.mkOption {
type = lib.types.path;
default = "/var/lib/nixos-options-mcp";
description = "Directory to store data files.";
};
database = {
type = lib.mkOption {
type = lib.types.enum [ "sqlite" "postgres" ];
default = "sqlite";
description = "Database backend to use.";
};
name = lib.mkOption {
type = lib.types.str;
default = "nixos-options.db";
description = "SQLite database filename (when using sqlite backend).";
};
connectionString = lib.mkOption {
type = lib.types.str;
default = "";
description = ''
PostgreSQL connection string (when using postgres backend).
Example: "postgres://user:password@localhost/nixos_options?sslmode=disable"
WARNING: This value will be stored in the Nix store, which is world-readable.
For production use with sensitive credentials, use connectionStringFile instead.
'';
};
connectionStringFile = lib.mkOption {
type = lib.types.nullOr lib.types.path;
default = null;
description = ''
Path to a file containing the PostgreSQL connection string.
The file should contain just the connection string, e.g.:
postgres://user:password@localhost/nixos_options?sslmode=disable
This is the recommended way to configure PostgreSQL credentials
as the file is not stored in the world-readable Nix store.
The file must be readable by the service user.
'';
example = "/run/secrets/nixos-options-mcp-db";
};
};
indexOnStart = lib.mkOption {
type = lib.types.listOf lib.types.str;
default = [ ];
example = [ "nixos-unstable" "nixos-24.11" ];
description = ''
List of nixpkgs revisions to index on service start.
Can be channel names (nixos-unstable) or git hashes.
Indexing is skipped if the revision is already indexed.
'';
};
openFirewall = lib.mkOption {
type = lib.types.bool;
default = false;
description = ''
Whether to open the firewall for the MCP server.
Note: MCP typically runs over stdio, so this is usually not needed.
'';
};
};
config = lib.mkIf cfg.enable {
assertions = [
{
assertion = cfg.database.type == "sqlite"
|| cfg.database.connectionString != ""
|| cfg.database.connectionStringFile != null;
message = "services.nixos-options-mcp.database: when using postgres backend, either connectionString or connectionStringFile must be set";
}
{
assertion = cfg.database.connectionString == "" || cfg.database.connectionStringFile == null;
message = "services.nixos-options-mcp.database: connectionString and connectionStringFile are mutually exclusive";
}
];
users.users.${cfg.user} = lib.mkIf (cfg.user == "nixos-options-mcp") {
isSystemUser = true;
group = cfg.group;
home = cfg.dataDir;
description = "NixOS Options MCP server user";
};
users.groups.${cfg.group} = lib.mkIf (cfg.group == "nixos-options-mcp") { };
systemd.tmpfiles.rules = [
"d ${cfg.dataDir} 0750 ${cfg.user} ${cfg.group} -"
];
systemd.services.nixos-options-mcp = {
description = "NixOS Options MCP Server";
wantedBy = [ "multi-user.target" ];
after = [ "network.target" ]
++ lib.optional (cfg.database.type == "postgres") "postgresql.service";
environment = lib.mkIf (!useConnectionStringFile) {
NIXOS_OPTIONS_DATABASE = databaseUrl;
};
path = [ cfg.package ];
script = let
indexCommands = lib.optionalString (cfg.indexOnStart != []) ''
${lib.concatMapStringsSep "\n" (rev: ''
echo "Indexing revision: ${rev}"
nixos-options index "${rev}" || true
'') cfg.indexOnStart}
'';
in
if useConnectionStringFile then ''
# Read database connection string from file
if [ ! -f "${cfg.database.connectionStringFile}" ]; then
echo "Error: connectionStringFile not found: ${cfg.database.connectionStringFile}" >&2
exit 1
fi
export NIXOS_OPTIONS_DATABASE="$(cat "${cfg.database.connectionStringFile}")"
${indexCommands}
exec nixos-options serve
'' else ''
${indexCommands}
exec nixos-options serve
'';
serviceConfig = {
Type = "simple";
User = cfg.user;
Group = cfg.group;
Restart = "on-failure";
RestartSec = "5s";
# Hardening
NoNewPrivileges = true;
ProtectSystem = "strict";
ProtectHome = true;
PrivateTmp = true;
PrivateDevices = true;
ProtectKernelTunables = true;
ProtectKernelModules = true;
ProtectControlGroups = true;
RestrictNamespaces = true;
RestrictRealtime = true;
RestrictSUIDSGID = true;
MemoryDenyWriteExecute = true;
LockPersonality = true;
ReadWritePaths = [ cfg.dataDir ];
WorkingDirectory = cfg.dataDir;
StateDirectory = lib.mkIf (cfg.dataDir == "/var/lib/nixos-options-mcp") "nixos-options-mcp";
};
};
};
}

26
nix/package.nix Normal file
View File

@@ -0,0 +1,26 @@
{ lib, buildGoModule, makeWrapper, nix, src }:
buildGoModule {
pname = "nixos-options-mcp";
version = "0.1.0";
inherit src;
vendorHash = "sha256-D0KIxQC9ctIAaHBFTvkhBE06uOZwDUcIw8471Ug2doY=";
subPackages = [ "cmd/nixos-options" ];
nativeBuildInputs = [ makeWrapper ];
postInstall = ''
wrapProgram $out/bin/nixos-options \
--prefix PATH : ${lib.makeBinPath [ nix ]}
'';
meta = with lib; {
description = "MCP server for NixOS options search and query";
homepage = "https://git.t-juice.club/torjus/labmcp";
license = licenses.mit;
maintainers = [ ];
mainProgram = "nixos-options";
};
}

107
testdata/options-sample.json vendored Normal file
View File

@@ -0,0 +1,107 @@
{
"services.nginx.enable": {
"declarations": ["/nix/store/xxx-source/nixos/modules/services/web-servers/nginx/default.nix"],
"default": false,
"description": "Whether to enable Nginx Web Server.",
"readOnly": false,
"type": "boolean"
},
"services.nginx.package": {
"declarations": ["/nix/store/xxx-source/nixos/modules/services/web-servers/nginx/default.nix"],
"default": {},
"description": {"_type": "mdDoc", "text": "The nginx package to use."},
"readOnly": false,
"type": "package"
},
"services.nginx.user": {
"declarations": ["/nix/store/xxx-source/nixos/modules/services/web-servers/nginx/default.nix"],
"default": "nginx",
"description": "User account under which nginx runs.",
"readOnly": false,
"type": "string"
},
"services.nginx.group": {
"declarations": ["/nix/store/xxx-source/nixos/modules/services/web-servers/nginx/default.nix"],
"default": "nginx",
"description": "Group account under which nginx runs.",
"readOnly": false,
"type": "string"
},
"services.nginx.virtualHosts": {
"declarations": ["/nix/store/xxx-source/nixos/modules/services/web-servers/nginx/default.nix"],
"default": {},
"description": "Declarative vhost configuration.",
"readOnly": false,
"type": "attrsOf (submodule)"
},
"services.caddy.enable": {
"declarations": ["/nix/store/xxx-source/nixos/modules/services/web-servers/caddy/default.nix"],
"default": false,
"description": "Enable the Caddy web server.",
"readOnly": false,
"type": "boolean"
},
"services.caddy.package": {
"declarations": ["/nix/store/xxx-source/nixos/modules/services/web-servers/caddy/default.nix"],
"default": {},
"description": "Caddy package to use.",
"readOnly": false,
"type": "package"
},
"programs.git.enable": {
"declarations": ["/nix/store/xxx-source/nixos/modules/programs/git.nix"],
"default": false,
"description": "Whether to enable git.",
"readOnly": false,
"type": "boolean"
},
"programs.git.package": {
"declarations": ["/nix/store/xxx-source/nixos/modules/programs/git.nix"],
"default": {},
"description": "The git package to use.",
"readOnly": false,
"type": "package"
},
"programs.git.config": {
"declarations": ["/nix/store/xxx-source/nixos/modules/programs/git.nix"],
"default": {},
"description": "Git configuration options.",
"readOnly": false,
"type": "attrsOf (attrsOf anything)"
},
"networking.firewall.enable": {
"declarations": ["/nix/store/xxx-source/nixos/modules/services/networking/firewall.nix"],
"default": true,
"description": "Whether to enable the firewall.",
"readOnly": false,
"type": "boolean"
},
"networking.firewall.allowedTCPPorts": {
"declarations": ["/nix/store/xxx-source/nixos/modules/services/networking/firewall.nix"],
"default": [],
"description": "List of TCP ports to allow through the firewall.",
"readOnly": false,
"type": "listOf port"
},
"networking.firewall.allowedUDPPorts": {
"declarations": ["/nix/store/xxx-source/nixos/modules/services/networking/firewall.nix"],
"default": [],
"description": "List of UDP ports to allow through the firewall.",
"readOnly": false,
"type": "listOf port"
},
"boot.loader.grub.enable": {
"declarations": ["/nix/store/xxx-source/nixos/modules/system/boot/loader/grub/grub.nix"],
"default": true,
"description": "Whether to enable GRUB.",
"readOnly": false,
"type": "boolean"
},
"boot.loader.grub.device": {
"declarations": ["/nix/store/xxx-source/nixos/modules/system/boot/loader/grub/grub.nix"],
"default": "",
"description": "The device where GRUB will be installed.",
"readOnly": false,
"type": "string"
}
}