bulk commit changes!
This commit is contained in:
@@ -0,0 +1,2 @@
|
||||
schema: spec-driven
|
||||
created: 2026-02-12
|
||||
@@ -0,0 +1,111 @@
|
||||
## Context
|
||||
|
||||
ClawFort needs a stunning one-page placeholder website that automatically aggregates and displays AI news hourly. The site must be containerized, use Perplexity API for news generation, and feature infinite scroll with 30-day retention.
|
||||
|
||||
**Current State:** Greenfield project - no existing codebase.
|
||||
|
||||
**Constraints:**
|
||||
- Must use Perplexity API (API key via environment variable)
|
||||
- Containerized deployment (Docker)
|
||||
- Lean JavaScript framework for frontend
|
||||
- 30-day news retention with archiving
|
||||
- Hourly automated updates
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
- Stunning one-page website with ClawFort branding
|
||||
- Hourly AI news aggregation via Perplexity API
|
||||
- Dynamic hero block with featured news and image
|
||||
- Infinite scroll news feed (10 initial items)
|
||||
- 30-day retention with automatic archiving
|
||||
- Source attribution for news and images
|
||||
- Fully containerized deployment
|
||||
- Responsive design
|
||||
|
||||
**Non-Goals:**
|
||||
- User authentication/accounts
|
||||
- Manual news curation interface
|
||||
- Real-time updates (polling only)
|
||||
- Multi-language support
|
||||
- CMS integration
|
||||
- SEO optimization beyond basic meta tags
|
||||
|
||||
## Decisions
|
||||
|
||||
### Architecture: Monolithic Container
|
||||
**Decision:** Single container with frontend + backend + SQLite
|
||||
**Rationale:** Simplicity for a placeholder site, easy deployment, no external database dependencies
|
||||
**Alternative:** Microservices with separate DB container - rejected as overkill for this scope
|
||||
|
||||
### Frontend Framework: Alpine.js + Tailwind CSS
|
||||
**Decision:** Alpine.js for lean reactivity, Tailwind for styling
|
||||
**Rationale:** Minimal bundle size (~15kb), no build step complexity, perfect for one-page sites
|
||||
**Alternative:** React/Vue - rejected as too heavy for simple infinite scroll and hero display
|
||||
|
||||
### Backend: Python (FastAPI) + APScheduler
|
||||
**Decision:** FastAPI for REST API, APScheduler for cron-like jobs
|
||||
**Rationale:** Fast to develop, excellent async support, built-in OpenAPI docs, simple scheduling
|
||||
**Alternative:** Node.js/Express - rejected; Python better for data processing and Perplexity integration
|
||||
|
||||
### Database: SQLite with SQLAlchemy
|
||||
**Decision:** SQLite for zero-config persistence
|
||||
**Rationale:** No separate DB container needed, sufficient for 30-day news retention (~1000-2000 records)
|
||||
**Alternative:** PostgreSQL - rejected as adds deployment complexity
|
||||
|
||||
### News Aggregation Strategy
|
||||
**Decision:** Hourly cron job queries Perplexity for "latest AI news" with image generation
|
||||
**Rationale:** Simple, reliable, cost-effective
|
||||
**Implementation:**
|
||||
- Perplexity API call: "What are the latest AI news from the last hour?"
|
||||
- Store: headline, summary, source URL, image URL, timestamp
|
||||
- Attribution: Display source name and image credit
|
||||
|
||||
### Image Strategy
|
||||
**Decision:** Use Perplexity to suggest relevant images or generate via DALL-E if available, with local image optimization
|
||||
**Rationale:** Consistent with AI theme, no copyright concerns, plus configurable compression
|
||||
**Implementation:**
|
||||
- Download and optimize images locally using Pillow
|
||||
- Configurable quality setting via `IMAGE_QUALITY` env var (1-100, default 85)
|
||||
- Store optimized images in `/app/static/images/`
|
||||
- Serve optimized versions, fallback to original URL if optimization fails
|
||||
**Alternative:** Unsplash API - rejected to keep dependencies minimal
|
||||
|
||||
### Infinite Scroll Implementation
|
||||
**Decision:** Cursor-based pagination with Intersection Observer API
|
||||
**Rationale:** Efficient for large datasets, simple Alpine.js integration
|
||||
**Page Size:** 10 items per request
|
||||
|
||||
### Archive Strategy
|
||||
**Decision:** Soft delete (archived flag) + nightly cleanup job
|
||||
**Rationale:** Easy to implement, data recoverable if needed
|
||||
**Cleanup:** Move items >30 days to archive table or delete
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
**[Risk] Perplexity API rate limits or downtime** → Mitigation: Implement exponential backoff, cache last successful fetch, display cached content with "last updated" timestamp, fallback to OpenRouter API if configured
|
||||
|
||||
**[Risk] Container storage grows unbounded** → Mitigation: SQLite WAL mode, volume mounts for persistence, 30-day hard limit on retention
|
||||
|
||||
**[Risk] News quality varies** → Mitigation: Basic filtering (require title + summary), manual blacklist capability in config
|
||||
|
||||
**[Risk] Cold start performance** → Mitigation: SQLite connection pooling, frontend CDN-ready static assets
|
||||
|
||||
**[Trade-off] SQLite vs PostgreSQL** → SQLite limits concurrent writes but acceptable for read-heavy news site
|
||||
|
||||
**[Trade-off] Single container vs microservices** → Easier deployment but less scalable; acceptable for placeholder site
|
||||
|
||||
## Migration Plan
|
||||
|
||||
1. **Development:** Local Docker Compose setup
|
||||
2. **Environment:** Configure `PERPLEXITY_API_KEY` in `.env`
|
||||
3. **Build:** `docker build -t clawfort-site .`
|
||||
4. **Run:** `docker run -e PERPLEXITY_API_KEY=xxx -p 8000:8000 clawfort-site`
|
||||
5. **Data:** SQLite volume mount for persistence across restarts
|
||||
|
||||
## Open Questions (Resolved)
|
||||
|
||||
1. **Admin panel?** → Deferred to future
|
||||
2. **Image optimization?** → Yes, local optimization with Pillow, configurable quality via `IMAGE_QUALITY` env var
|
||||
3. **Analytics?** → Umami integration with `UMAMI_SCRIPT_URL` and `UMAMI_WEBSITE_ID` env vars, track page views, scroll events, and CTA clicks
|
||||
4. **API cost monitoring?** → Log Perplexity usage, fallback to OpenRouter API if `OPENROUTER_API_KEY` configured
|
||||
@@ -0,0 +1,54 @@
|
||||
## Why
|
||||
|
||||
ClawFort needs a stunning one-page placeholder website that automatically generates and displays AI news hourly, creating a dynamic, always-fresh brand presence without manual content curation. The site will serve as a living showcase of AI capabilities while building brand recognition.
|
||||
|
||||
## What Changes
|
||||
|
||||
- **New Capabilities:**
|
||||
- Automated AI news aggregation via Perplexity API (hourly updates)
|
||||
- Dynamic hero section with featured news and images
|
||||
- Infinite scroll news feed with 1-month retention
|
||||
- Archive system for older news items
|
||||
- Containerized deployment (Docker)
|
||||
- Responsive single-page design with lean JavaScript framework
|
||||
|
||||
- **Frontend:**
|
||||
- One-page website with hero block
|
||||
- Infinite scroll news feed (latest 10 on load)
|
||||
- News attribution to sources
|
||||
- Image credit display
|
||||
- Responsive design
|
||||
|
||||
- **Backend:**
|
||||
- News aggregation service (hourly cron job)
|
||||
- Perplexity API integration
|
||||
- News storage with 30-day retention
|
||||
- Archive management
|
||||
- REST API for frontend
|
||||
|
||||
- **Infrastructure:**
|
||||
- Docker containerization
|
||||
- Environment-based configuration
|
||||
- Perplexity API key management
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
- `news-aggregator`: Automated AI news collection via Perplexity API with hourly scheduling
|
||||
- `news-storage`: Database storage with 30-day retention and archive management
|
||||
- `hero-display`: Dynamic hero block with featured news and image attribution
|
||||
- `infinite-scroll`: Frontend infinite scroll with lazy loading (10 initial, paginated)
|
||||
- `containerized-deployment`: Docker-based deployment with environment configuration
|
||||
- `responsive-frontend`: Single-page application with lean JavaScript framework
|
||||
|
||||
### Modified Capabilities
|
||||
- None (new project)
|
||||
|
||||
## Impact
|
||||
|
||||
- **Code:** New full-stack application (frontend + backend)
|
||||
- **APIs:** Perplexity API integration required
|
||||
- **Dependencies:** Docker, Node.js/Python runtime, database (SQLite/PostgreSQL)
|
||||
- **Infrastructure:** Container orchestration support
|
||||
- **Environment:** `PERPLEXITY_API_KEY` required
|
||||
- **Data:** 30-day rolling news archive with automatic cleanup
|
||||
@@ -0,0 +1,45 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Containerized deployment
|
||||
The system SHALL run entirely within Docker containers with all dependencies included.
|
||||
|
||||
#### Scenario: Single container build
|
||||
- **WHEN** building the Docker image
|
||||
- **THEN** the Dockerfile SHALL include Python runtime, Node.js (for Tailwind if needed), and all application code
|
||||
- **AND** expose port 8000 for web traffic
|
||||
|
||||
#### Scenario: Environment configuration
|
||||
- **WHEN** running the container
|
||||
- **THEN** the system SHALL read PERPLEXITY_API_KEY from environment variables
|
||||
- **AND** fail to start if the key is missing or invalid
|
||||
- **AND** support optional configuration for retention days (default: 30)
|
||||
- **AND** support optional IMAGE_QUALITY for image compression (default: 85)
|
||||
- **AND** support optional OPENROUTER_API_KEY for fallback LLM provider
|
||||
- **AND** support optional UMAMI_SCRIPT_URL and UMAMI_WEBSITE_ID for analytics
|
||||
|
||||
#### Scenario: Data persistence
|
||||
- **WHEN** the container restarts
|
||||
- **THEN** the SQLite database SHALL persist via Docker volume mount
|
||||
- **AND** news data SHALL remain intact across restarts
|
||||
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Responsive single-page design
|
||||
The system SHALL provide a stunning, responsive one-page website with ClawFort branding.
|
||||
|
||||
#### Scenario: Brand consistency
|
||||
- **WHEN** viewing the website
|
||||
- **THEN** the design SHALL feature ClawFort branding (logo, colors, typography)
|
||||
- **AND** maintain visual consistency across all sections
|
||||
|
||||
#### Scenario: Responsive layout
|
||||
- **WHEN** viewing on mobile, tablet, or desktop
|
||||
- **THEN** the layout SHALL adapt appropriately
|
||||
- **AND** the hero block SHALL resize proportionally
|
||||
- **AND** the news feed SHALL use appropriate column layouts
|
||||
|
||||
#### Scenario: Performance
|
||||
- **WHEN** loading the page
|
||||
- **THEN** initial page load SHALL complete within 2 seconds
|
||||
- **AND** images SHALL lazy load outside viewport
|
||||
- **AND** JavaScript bundle SHALL be under 100KB gzipped
|
||||
@@ -0,0 +1,55 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Hero block display
|
||||
The system SHALL display the most recent news item as a featured hero block with full attribution.
|
||||
|
||||
#### Scenario: Hero rendering
|
||||
- **WHEN** the page loads
|
||||
- **THEN** the hero block SHALL display the latest news headline, summary, and featured image
|
||||
- **AND** show source attribution (e.g., "Via: TechCrunch")
|
||||
- **AND** show image credit (e.g., "Image: DALL-E")
|
||||
|
||||
#### Scenario: Hero update
|
||||
- **WHEN** new news is fetched hourly
|
||||
- **THEN** the hero block SHALL automatically update to show the newest item
|
||||
- **AND** the previous hero item SHALL move to the news feed
|
||||
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Infinite scroll news feed
|
||||
The system SHALL display news items in reverse chronological order with infinite scroll pagination.
|
||||
|
||||
#### Scenario: Initial load
|
||||
- **WHEN** the page first loads
|
||||
- **THEN** the system SHALL display the 10 most recent non-archived news items
|
||||
- **AND** exclude the hero item from the feed
|
||||
|
||||
#### Scenario: Infinite scroll
|
||||
- **WHEN** the user scrolls to the bottom of the feed
|
||||
- **THEN** the system SHALL fetch the next 10 news items via API
|
||||
- **AND** append them to the feed without page reload
|
||||
- **AND** show a loading indicator during fetch
|
||||
|
||||
#### Scenario: End of feed
|
||||
- **WHEN** all non-archived news items have been loaded
|
||||
- **THEN** the system SHALL display "No more news" message
|
||||
- **AND** disable further scroll triggers
|
||||
|
||||
### Requirement: News attribution display
|
||||
The system SHALL clearly attribute all news content and images to their sources.
|
||||
|
||||
#### Scenario: Source attribution
|
||||
- **WHEN** displaying any news item
|
||||
- **THEN** the system SHALL show the original source name and link
|
||||
- **AND** display image credit if available
|
||||
|
||||
#### Scenario: Perplexity attribution
|
||||
- **WHEN** displaying aggregated content
|
||||
- **THEN** the system SHALL include "Powered by Perplexity" in the footer
|
||||
|
||||
#### Scenario: Analytics tracking
|
||||
- **WHEN** Umami analytics is configured via `UMAMI_SCRIPT_URL` and `UMAMI_WEBSITE_ID`
|
||||
- **THEN** the system SHALL inject Umami tracking script into page head
|
||||
- **AND** track page view events on initial load
|
||||
- **AND** track scroll depth events (25%, 50%, 75%, 100%)
|
||||
- **AND** track CTA click events (news item clicks, source link clicks)
|
||||
@@ -0,0 +1,56 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: News aggregation via Perplexity API
|
||||
The system SHALL fetch AI news hourly from Perplexity API and store it with full attribution.
|
||||
|
||||
#### Scenario: Hourly news fetch
|
||||
- **WHEN** the scheduled job runs every hour
|
||||
- **THEN** the system calls Perplexity API with query "latest AI news"
|
||||
- **AND** stores the response with headline, summary, source URL, and timestamp
|
||||
|
||||
#### Scenario: API error handling
|
||||
- **WHEN** Perplexity API returns an error or timeout
|
||||
- **THEN** the system logs the error with cost tracking
|
||||
- **AND** retries with exponential backoff up to 3 times
|
||||
- **AND** falls back to OpenRouter API if `OPENROUTER_API_KEY` is configured
|
||||
- **AND** continues using cached content if all retries and fallback fail
|
||||
|
||||
### Requirement: Featured image generation
|
||||
The system SHALL generate or fetch a relevant featured image for each news item.
|
||||
|
||||
#### Scenario: Image acquisition
|
||||
- **WHEN** a new news item is fetched
|
||||
- **THEN** the system SHALL request a relevant image URL from Perplexity
|
||||
- **AND** download and optimize the image locally using Pillow
|
||||
- **AND** apply quality compression based on `IMAGE_QUALITY` env var (1-100, default 85)
|
||||
- **AND** store the optimized image path and original image credit/source information
|
||||
|
||||
#### Scenario: Image optimization configuration
|
||||
- **WHEN** the system processes an image
|
||||
- **THEN** it SHALL read `IMAGE_QUALITY` from environment (default: 85)
|
||||
- **AND** apply JPEG compression at specified quality level
|
||||
- **AND** resize images exceeding 1200px width while maintaining aspect ratio
|
||||
- **AND** store optimized images in `/app/static/images/` directory
|
||||
|
||||
#### Scenario: Image fallback
|
||||
- **WHEN** image generation fails or returns no result
|
||||
- **THEN** the system SHALL use a default ClawFort branded placeholder image
|
||||
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: News data persistence with retention
|
||||
The system SHALL store news items for exactly 30 days with automatic archiving.
|
||||
|
||||
#### Scenario: News storage
|
||||
- **WHEN** a news item is fetched from Perplexity
|
||||
- **THEN** the system SHALL store it in SQLite with fields: id, headline, summary, source_url, image_url, image_credit, published_at, created_at
|
||||
- **AND** set archived=false by default
|
||||
|
||||
#### Scenario: Automatic archiving
|
||||
- **WHEN** a nightly cleanup job runs
|
||||
- **THEN** the system SHALL mark all news items older than 30 days as archived=true
|
||||
- **AND** delete archived items older than 60 days permanently
|
||||
|
||||
#### Scenario: Duplicate prevention
|
||||
- **WHEN** fetching news that matches an existing headline (within 24 hours)
|
||||
- **THEN** the system SHALL skip insertion to prevent duplicates
|
||||
@@ -0,0 +1,82 @@
|
||||
## 1. Project Setup
|
||||
|
||||
- [x] 1.1 Create project directory structure (backend/, frontend/, docker/)
|
||||
- [x] 1.2 Initialize Python project with pyproject.toml (FastAPI, SQLAlchemy, APScheduler, httpx)
|
||||
- [x] 1.3 Create requirements.txt for Docker build
|
||||
- [x] 1.4 Set up Tailwind CSS configuration
|
||||
- [x] 1.5 Create .env.example with all environment variables (PERPLEXITY_API_KEY, IMAGE_QUALITY, OPENROUTER_API_KEY, UMAMI_SCRIPT_URL, UMAMI_WEBSITE_ID)
|
||||
|
||||
## 2. Database Layer
|
||||
|
||||
- [x] 2.1 Create SQLAlchemy models (NewsItem with fields: id, headline, summary, source_url, image_url, image_credit, published_at, created_at, archived)
|
||||
- [x] 2.2 Create database initialization and migration scripts
|
||||
- [x] 2.3 Implement database connection management with SQLite
|
||||
- [x] 2.4 Create repository functions (create_news, get_recent_news, get_news_paginated, archive_old_news, delete_archived_news)
|
||||
|
||||
## 3. News Aggregation Service
|
||||
|
||||
- [x] 3.1 Implement Perplexity API client with httpx and cost logging
|
||||
- [x] 3.2 Create news fetch function with query "latest AI news"
|
||||
- [x] 3.3 Implement exponential backoff retry logic (3 attempts)
|
||||
- [x] 3.4 Add duplicate detection (headline match within 24h)
|
||||
- [x] 3.5 Create hourly scheduled job with APScheduler
|
||||
- [x] 3.6 Implement image URL fetching from Perplexity
|
||||
- [x] 3.7 Add image download and optimization with Pillow (configurable quality)
|
||||
- [x] 3.8 Implement OpenRouter API fallback for news fetching
|
||||
- [x] 3.9 Add default placeholder image fallback
|
||||
|
||||
## 4. Backend API
|
||||
|
||||
- [x] 4.1 Create FastAPI application structure
|
||||
- [x] 4.2 Implement GET /api/news endpoint with pagination (cursor-based)
|
||||
- [x] 4.3 Implement GET /api/news/latest endpoint for hero block
|
||||
- [x] 4.4 Add CORS middleware for frontend access
|
||||
- [x] 4.5 Create Pydantic schemas for API responses
|
||||
- [x] 4.6 Implement health check endpoint
|
||||
- [x] 4.7 Add API error handling and logging
|
||||
|
||||
## 5. Frontend Implementation
|
||||
|
||||
- [x] 5.1 Create HTML structure with ClawFort branding
|
||||
- [x] 5.2 Implement hero block with Alpine.js (latest news display)
|
||||
- [x] 5.3 Create news feed component with Alpine.js
|
||||
- [x] 5.4 Implement infinite scroll with Intersection Observer API
|
||||
- [x] 5.5 Add loading indicators and "No more news" message
|
||||
- [x] 5.6 Implement source attribution display
|
||||
- [x] 5.7 Add image lazy loading
|
||||
- [x] 5.8 Style with Tailwind CSS (responsive design)
|
||||
- [x] 5.9 Add "Powered by Perplexity" footer attribution
|
||||
- [x] 5.10 Implement Umami analytics integration (conditional on env vars)
|
||||
- [x] 5.11 Add analytics events: page view, scroll depth (25/50/75/100%), CTA clicks
|
||||
|
||||
## 6. Archive Management
|
||||
|
||||
- [x] 6.1 Implement nightly cleanup job (archive >30 days)
|
||||
- [x] 6.2 Create permanent deletion job (>60 days archived)
|
||||
- [x] 6.3 Add retention configuration (default 30 days)
|
||||
|
||||
## 7. Docker Containerization
|
||||
|
||||
- [x] 7.1 Create Dockerfile with multi-stage build (Python + static assets)
|
||||
- [x] 7.2 Create docker-compose.yml for local development
|
||||
- [x] 7.3 Add volume mount for SQLite persistence
|
||||
- [x] 7.4 Configure environment variable handling
|
||||
- [x] 7.5 Optimize image size (slim Python base)
|
||||
- [x] 7.6 Add .dockerignore file
|
||||
|
||||
## 8. Testing & Validation
|
||||
|
||||
- [x] 8.1 Test Perplexity API integration manually
|
||||
- [x] 8.2 Verify hourly news fetching works
|
||||
- [x] 8.3 Test infinite scroll pagination
|
||||
- [x] 8.4 Verify responsive design on mobile/desktop
|
||||
- [x] 8.5 Test container build and run
|
||||
- [x] 8.6 Verify data persistence across container restarts
|
||||
- [x] 8.7 Test archive cleanup functionality
|
||||
|
||||
## 9. Documentation
|
||||
|
||||
- [x] 9.1 Create README.md with setup instructions
|
||||
- [x] 9.2 Document environment variables
|
||||
- [x] 9.3 Add deployment instructions
|
||||
- [x] 9.4 Document API endpoints
|
||||
Reference in New Issue
Block a user