News & Publications Maintenance Guide

This document captures the moving parts behind the hybrid News and Publications experience so future updates stay safe, accessible, and SEO-friendly.

Data Flow Overview

1. Source fan-out (lib/sources/)
   Each module fetches raw records (RSS, Crossref, OpenAlex, CourtListener). Update URLs, query params, or auth headers here.
2. Normalization (lib/content/normalize.ts)
   Transforms upstream payloads into FeedItem objects with stable IDs (lib/content/hash.ts).
3. Ranking (lib/content/rank.ts)
   Scores items based on recency, source authority, query hits, and type boosts. Tweaking weights affects both SSE output and pre-rendered tag pages.
4. Caching (lib/content/cache.ts)
   Lightweight in-memory cache with 15-minute TTL. Use only for short-lived warm caches (no persistence).
5. Streaming (server/index.ts)
   Express SSE endpoints (/api/news, /api/pubs) orchestrate fan-out, dedupe items, enrich with citations/why-text, and stream events.
6. Client enhancement (public/components/TerminalFeed.js)
   Manages the live terminal, card rendering, pinning, clipboard actions, analytics hooks, and toasts.

Adding or Updating Feeds

1. Implement fetch logic inside lib/sources/*.ts. Keep network requests lightweight and respect source rate limits.
2. Update lib/sources/index.ts to add the new fan-out promise.
3. If the source requires custom parsing, extend lib/content/normalize.ts.
4. Keep the default query focused on CEQA/NEPA + AI (("CEQA" OR "California Environmental Quality Act" OR "NEPA" OR "National Environmental Policy Act") AND ("artificial intelligence" OR "AI" OR "machine learning")) unless a page explicitly overrides it.
5. Run npm run build:ts followed by npm test to ensure the build succeeds.
6. Smoke test /news or /publications in live mode.

Tuning Ranking

• Adjust authority weights in AUTHORITY (inside lib/content/rank.ts).
• Modify the recency calculation to change how quickly older items drop in priority.
• Use the optional type parameter in SSE requests to boost specific feeds in custom clients.

Tag Pages

• Tag metadata lives in _data/newsTags.js and _data/publicationTags.js. Each entry needs a slug, title, description, and query.
• Tag templates (news/tag.11ty.js, publications/tag.11ty.js) call fetchTopItems during the Eleventy build. The helper composes the same enrichment pipeline used by SSE responses.
• Adding new tags requires running npm run build:ts before starting Eleventy so build/lib/server/fetchTop.js exists.

Terminal Client & Actions

• Edit the TypeScript source in components/TerminalFeed.ts. Compile with npm run build:components.
• Buttons emit analytics events via window.ceqaAnalytics?.emit. If an analytics provider is later integrated, attach an adapter to window.ceqaAnalytics.
• To tweak copy toasts or card markup, update both the TS source and the compiled JS if you do not run the compiler.

Library Page

• Client logic lives in public/assets/scripts/library.js. It relies on the same citation helpers (public/assets/scripts/citations.js) and why-text helper (public/assets/scripts/why.js).
• Pins are stored in localStorage under the ceqa_library key; clearing or exporting operates entirely client-side.

Testing & Deployment Checklist

1. npm run test
2. npm run build
3. Open /news → confirm the live stream loads and the Refresh button pulls a new batch.
4. Open /publications → confirm the live stream loads and the Refresh button pulls a new batch.
5. Visit /news/tag/... and /publications/tag/....
6. Pin an item and confirm it appears on /library.
7. Verify JSON/CSV exports download correctly.

Following this playbook helps ensure the hybrid experience stays reliable, crawlable, and easy to maintain.