diff --git a/.github/scripts/upgrade-test/README.md b/.github/scripts/upgrade-test/README.md new file mode 100644 index 00000000..68fadc73 --- /dev/null +++ b/.github/scripts/upgrade-test/README.md @@ -0,0 +1,259 @@ +# HomeBox Upgrade Testing Workflow + +This document describes the automated upgrade testing workflow for HomeBox. + +## Overview + +The upgrade test workflow is designed to ensure data integrity and functionality when upgrading HomeBox from one version to another. It automatically: + +1. Deploys a stable version of HomeBox +2. Creates test data (users, items, locations, labels, notifiers, attachments) +3. Upgrades to the latest version from the main branch +4. Verifies all data and functionality remain intact + +## Workflow File + +**Location**: `.github/workflows/upgrade-test.yaml` + +## Trigger Conditions + +The workflow runs: +- **Daily**: Automatically at 2 AM UTC (via cron schedule) +- **Manual**: Can be triggered manually via GitHub Actions UI +- **On Push**: When changes are made to the workflow files or test scripts + +## Test Scenarios + +### 1. Environment Setup +- Pulls the latest stable HomeBox Docker image from GHCR +- Starts the application with test configuration +- Ensures the service is healthy and ready + +### 2. Data Creation + +The workflow creates comprehensive test data using the `create-test-data.sh` script: + +#### Users and Groups +- **Group 1**: 5 users (user1@homebox.test through user5@homebox.test) +- **Group 2**: 2 users (user6@homebox.test and user7@homebox.test) +- All users have password: `TestPassword123!` + +#### Locations +- **Group 1**: Living Room, Garage +- **Group 2**: Home Office + +#### Labels +- **Group 1**: Electronics, Important +- **Group 2**: Work Equipment + +#### Items +- **Group 1**: 5 items (Laptop Computer, Power Drill, TV Remote, Tool Box, Coffee Maker) +- **Group 2**: 2 items (Monitor, Keyboard) + +#### Attachments +- Multiple attachments added to various items (receipts, manuals, warranties) + +#### Notifiers +- **Group 1**: Test notifier named "TESTING" + +### 3. Upgrade Process + +1. Stops the stable version container +2. Builds a fresh image from the current main branch +3. Copies the database to a new location +4. Starts the new version with the existing data + +### 4. Verification Tests + +The Playwright test suite (`upgrade-verification.spec.ts`) verifies: + +- ✅ **User Authentication**: All 7 users can log in with their credentials +- ✅ **Data Persistence**: All items, locations, and labels are present +- ✅ **Attachments**: File attachments are correctly associated with items +- ✅ **Notifiers**: The "TESTING" notifier is still configured +- ✅ **UI Functionality**: Version display, theme switching work correctly +- ✅ **Data Isolation**: Groups can only see their own data + +## Test Data File + +The setup script generates a JSON file at `/tmp/test-users.json` containing: + +```json +{ + "users": [ + { + "email": "user1@homebox.test", + "password": "TestPassword123!", + "token": "...", + "group": "1" + }, + ... + ], + "locations": { + "group1": ["location-id-1", "location-id-2"], + "group2": ["location-id-3"] + }, + "labels": {...}, + "items": {...}, + "notifiers": {...} +} +``` + +This file is used by the Playwright tests to verify data integrity. + +## Scripts + +### create-test-data.sh + +**Location**: `.github/scripts/upgrade-test/create-test-data.sh` + +**Purpose**: Creates all test data via the HomeBox REST API + +**Environment Variables**: +- `HOMEBOX_URL`: Base URL of the HomeBox instance (default: http://localhost:7745) +- `TEST_DATA_FILE`: Path to output JSON file (default: /tmp/test-users.json) + +**Requirements**: +- `curl`: For API calls +- `jq`: For JSON processing + +**Usage**: +```bash +export HOMEBOX_URL=http://localhost:7745 +./.github/scripts/upgrade-test/create-test-data.sh +``` + +## Running Tests Locally + +To run the upgrade tests locally: + +### Prerequisites +```bash +# Install dependencies +sudo apt-get install -y jq curl docker.io + +# Install pnpm and Playwright +cd frontend +pnpm install +pnpm exec playwright install --with-deps chromium +``` + +### Run the test +```bash +# Start stable version +docker run -d \ + --name homebox-test \ + -p 7745:7745 \ + -e HBOX_OPTIONS_ALLOW_REGISTRATION=true \ + -v /tmp/homebox-data:/data \ + ghcr.io/sysadminsmedia/homebox:latest + +# Wait for startup +sleep 10 + +# Create test data +export HOMEBOX_URL=http://localhost:7745 +./.github/scripts/upgrade-test/create-test-data.sh + +# Stop container +docker stop homebox-test +docker rm homebox-test + +# Build new version +docker build -t homebox:test . + +# Start new version with existing data +docker run -d \ + --name homebox-test \ + -p 7745:7745 \ + -e HBOX_OPTIONS_ALLOW_REGISTRATION=true \ + -v /tmp/homebox-data:/data \ + homebox:test + +# Wait for startup +sleep 10 + +# Run verification tests +cd frontend +TEST_DATA_FILE=/tmp/test-users.json \ +E2E_BASE_URL=http://localhost:7745 \ +pnpm exec playwright test \ + --project=chromium \ + test/e2e/upgrade-verification.spec.ts + +# Cleanup +docker stop homebox-test +docker rm homebox-test +``` + +## Artifacts + +The workflow produces several artifacts: + +1. **playwright-report-upgrade-test**: HTML report of test results +2. **playwright-traces**: Detailed traces for debugging failures +3. **Docker logs**: Collected on failure for troubleshooting + +## Failure Scenarios + +The workflow will fail if: +- The stable version fails to start +- Test data creation fails +- The new version fails to start with existing data +- Any verification test fails +- Database migrations fail + +## Troubleshooting + +### Test Data Creation Fails + +Check the Docker logs: +```bash +docker logs homebox-old +``` + +Verify the API is accessible: +```bash +curl http://localhost:7745/api/v1/status +``` + +### Verification Tests Fail + +1. Download the Playwright report from GitHub Actions artifacts +2. Review the HTML report for detailed failure information +3. Check traces for visual debugging + +### Database Issues + +If migrations fail: +```bash +# Check database file +ls -lh /tmp/homebox-data-new/homebox.db + +# Check Docker logs for migration errors +docker logs homebox-new +``` + +## Future Enhancements + +Potential improvements: +- [ ] Test multiple upgrade paths (e.g., v0.10 → v0.11 → v0.12) +- [ ] Test with PostgreSQL backend in addition to SQLite +- [ ] Add performance benchmarks +- [ ] Test with larger datasets +- [ ] Add API-level verification in addition to UI tests +- [ ] Test backup and restore functionality + +## Related Files + +- `.github/workflows/upgrade-test.yaml` - Main workflow definition +- `.github/scripts/upgrade-test/create-test-data.sh` - Data generation script +- `frontend/test/e2e/upgrade-verification.spec.ts` - Playwright verification tests +- `.github/workflows/e2e-partial.yaml` - Standard E2E test workflow (for reference) + +## Support + +For issues or questions about this workflow: +1. Check the GitHub Actions run logs +2. Review this documentation +3. Open an issue in the repository