Federated from workspace · PRD-001 · Dental Clinic Revenue Operating System/docs/deployment/CUSTOMER_1_GO_LIVE_RUNBOOK.md Do not edit canonical truth here — update the source repo, then re-run npm run docs:sync.

Customer #1 — Production Go-Live Runbook

Full stack (company + clinic + docs): See the canonical Production Deployment Guide in the engineering docs portal (source: zaixos-engineering-platform/docs/DEPLOYMENT/production-deployment-guide.md).

Repository: Dental Clinic Revenue Operating System (PRD-001)
Audience: Developer / DevOps engineer
Scope: Production deployment readiness only — no product feature work
Authority: docs/deployment/PRODUCTION_RUNBOOK.md · PilotDeploymentValidationService · .env.example · bootstrap/app.php

GO definition: php artisan pilot:validate-deployment --strict exits 0 on the production host, manual smoke test (Section 10) passes, and clinic staff can complete lead → payment without developer intervention.

Assumed deploy path: /var/www/dental-clinic-ros (adjust paths consistently).

1. Production Server Requirements

Why required

Customer #1 runs a multi-tenant Laravel 11 monolith with PostgreSQL, Redis-backed cache/queue (production defaults), scheduled jobs, and Filament admin. Missing components cause silent failures (no reminders, stuck jobs, 500 on public site).

Stack

Component	Minimum	Recommended	Why
OS	Ubuntu 22.04 LTS	Ubuntu 24.04 LTS	Supported PHP/Postgres packages
CPU	2 vCPU	4 vCPU	Queue workers + PHP-FPM concurrency
RAM	4 GB	8 GB	Redis + Postgres + 2 workers
Disk	40 GB SSD	80 GB SSD	DB + `storage/` media + backups
PHP	8.2+	8.3	`composer.json` requires `^8.2`
PostgreSQL	15+	16	Production validation expects `pgsql`
Redis	7.x	7.x	Default `CACHE_STORE` / `QUEUE_CONNECTION` in production
Nginx	1.18+	latest stable	HTTPS termination, PHP-FPM proxy
Supervisor	4.x	4.x	Persistent `queue:work`
Cron	system cron	`www-data` user	`schedule:run` every minute
SSL	Let's Encrypt or provider cert	auto-renew	`APP_URL` must be `https://` in production

PHP extensions (required)

bash

sudo apt install -y php8.3-fpm php8.3-cli php8.3-pgsql php8.3-mbstring php8.3-xml \
  php8.3-curl php8.3-zip php8.3-bcmath php8.3-intl php8.3-redis

Optional: php8.3-gd or imagick (media thumbnails).

Initial server packages

bash

sudo apt update && sudo apt upgrade -y
sudo apt install -y nginx postgresql postgresql-contrib redis-server supervisor cron \
  git unzip curl
curl -sS https://getcomposer.org/installer | php
sudo mv composer.phar /usr/local/bin/composer

Verification

bash

php -v          # PHP 8.2+
psql --version  # PostgreSQL 15+
redis-cli ping  # PONG
nginx -v
supervisord --version

Expected: All commands succeed; redis-cli ping returns PONG.

Common failures: Redis not started → sudo systemctl enable --now redis-server.

2. Environment Configuration

Why required

Laravel loads .env at runtime. Production defaults in config/cache.php and config/queue.php use Redis when APP_ENV=production. Wrong values break mail, tenancy, sessions, and async jobs.

Files to configure

File	Purpose
`.env`	Machine-specific secrets and overrides
`config/app.php`	Fallback when env unset (`debug` defaults `false`)
`config/tenancy.php`	Tenant host resolution
`config/saas.php`	Email verification, billing
`config/clinic.php`	Reminders, automation flags
`config/ai.php`	Copilot provider
`config/backup.php`	Daily backups

After any .env change:

bash

php artisan config:clear
php artisan config:cache
php artisan route:cache
php artisan view:cache

Production `.env` template (Customer #1)

Copy from .env.example, then set at minimum:

dotenv

# --- Application ---
APP_NAME="Dental Clinic ROS"
APP_ENV=production
APP_KEY=base64:...                    # php artisan key:generate
APP_DEBUG=false
APP_URL=https://account.YOURBASEDOMAIN.com

# --- Database (PostgreSQL required in production) ---
DB_CONNECTION=pgsql
DB_HOST=127.0.0.1
DB_PORT=5432
DB_DATABASE=dental_clinic_ros
DB_USERNAME=dental_app
DB_PASSWORD=<strong-password>

# --- Session ---
SESSION_DRIVER=database
SESSION_LIFETIME=120
# SESSION_SECURE_COOKIE auto-true when APP_ENV=production (config/session.php)

# --- Cache & Queue (Redis recommended — matches production defaults) ---
CACHE_STORE=redis
QUEUE_CONNECTION=redis
REDIS_CLIENT=phpredis
REDIS_HOST=127.0.0.1
REDIS_PASSWORD=null
REDIS_PORT=6379

# --- Mail (required for verification + reminders) ---
MAIL_MAILER=smtp
MAIL_HOST=smtp.yourprovider.com
MAIL_PORT=587
MAIL_USERNAME=<smtp-user>
MAIL_PASSWORD=<smtp-password>
MAIL_ENCRYPTION=tls
MAIL_FROM_ADDRESS=noreply@YOURBASEDOMAIN.com
MAIL_FROM_NAME="${APP_NAME}"

# --- Filesystem ---
FILESYSTEM_DISK=local
# Optional S3: FILESYSTEM_DISK=s3 + AWS_* vars

# --- Tenancy ---
TENANCY_ENABLED=true
TENANCY_ENFORCE_USER_CLINIC=true
TENANCY_BASE_DOMAIN=YOURBASEDOMAIN.com
TENANCY_PUBLIC_DEFAULT_CLINIC_FALLBACK=false

# --- SaaS ---
SAAS_REQUIRE_EMAIL_VERIFICATION=true
SAAS_BILLING_PROVIDER=stripe
SAAS_PILOT_ENABLE_ALL_FEATURES=true
# SAAS_PILOT_CLINIC_IDS=1   # optional: restrict pilot entitlements

# --- Stripe (if billing enabled) ---
STRIPE_KEY=pk_live_...
STRIPE_SECRET=sk_live_...
STRIPE_WEBHOOK_SECRET=whsec_...

# --- Backups ---
BACKUP_ENABLED=true
BACKUP_DISK=local
BACKUP_PATH=backups
BACKUP_RETENTION_DAYS=30
BACKUP_SCHEDULE_TIME=02:00
# BACKUP_HANDLERS=App\\Modules\\Shared\\Infrastructure\\Backup\\S3BackupHandler

# --- Clinic automation (recommended OFF for first 7 days of pilot) ---
CLINIC_SCHEDULING_DEBUG=false
CLINIC_AUTOPILOT_ENABLED=false
CLINIC_AI_MANAGER_ENABLED=false
CLINIC_ORCHESTRATION_ENABLED=true
CLINIC_ORCHESTRATION_EXECUTE=false
CLINIC_ORCHESTRATION_EXECUTE_EVENTS=false
CLINIC_SMART_NOTIFICATIONS_ENABLED=true
CLINIC_DAILY_REPORT_ENABLED=true
CLINIC_DAILY_REPORT_SEND_EMAIL=false

# --- AI Copilot (optional day 1 — fallback rules work without LLM) ---
AI_DEFAULT_PROVIDER=groq
AI_GROQ_MODE=live
GROQ_API_KEY=<secret>
GROQ_BASE_URL=https://api.groq.com/openai/v1
AI_GROQ_MODEL=llama-3.3-70b-versatile
AI_COPILOT_PROVIDER=groq
AI_COPILOT_MODEL=llama-3.3-70b-versatile
CLINIC_COPILOT_AGENT_ENABLED=true
CLINIC_COPILOT_FALLBACK_RULES=true

# --- Logging ---
LOG_CHANNEL=stack
LOG_LEVEL=warning

Variable reference (project-used only)

Group	Variables	Production value
APP	`APP_ENV`, `APP_DEBUG`, `APP_URL`, `APP_KEY`	`production`, `false`, `https://...`, generated
DB	`DB_CONNECTION`, `DB_HOST`, `DB_PORT`, `DB_DATABASE`, `DB_USERNAME`, `DB_PASSWORD`	`pgsql`, ...
CACHE	`CACHE_STORE`, `REDIS_*`	`redis` + reachable Redis
QUEUE	`QUEUE_CONNECTION`, `REDIS_*`	`redis` (never `sync`)
SESSION	`SESSION_DRIVER`, `SESSION_LIFETIME`	`database` or `redis`
MAIL	`MAIL_MAILER`, `MAIL_HOST`, `MAIL_PORT`, `MAIL_USERNAME`, `MAIL_PASSWORD`, `MAIL_FROM_*`	Real SMTP
FILESYSTEM	`FILESYSTEM_DISK`, `AWS_*` (if S3)	`local` or `s3`
TENANCY	`TENANCY_ENABLED`, `TENANCY_BASE_DOMAIN`, `TENANCY_ENFORCE_USER_CLINIC`, `TENANCY_PUBLIC_DEFAULT_CLINIC_FALLBACK`	`true`, apex domain, `true`, `false`
SAAS	`SAAS_REQUIRE_EMAIL_VERIFICATION`, `SAAS_BILLING_PROVIDER`, `SAAS_PILOT_*`	`true`, `stripe`, as needed
AI	`AI_DEFAULT_PROVIDER`, `AI_GROQ_MODE`, `GROQ_API_KEY`, `AI_COPILOT_`, `CLINIC_COPILOT_`	Live provider or stub + fallback
Clinic flags	`CLINIC_AUTOPILOT_ENABLED`, `CLINIC_AI_MANAGER_ENABLED`, `CLINIC_ORCHESTRATION_`, `CLINIC__REMINDER_*`	See template above
BACKUP	`BACKUP_ENABLED`, `BACKUP_DISK`, `BACKUP_PATH`, `BACKUP_HANDLERS`	Enabled + off-site handler recommended

Verification

bash

php artisan zaixos:validate-env
php artisan config:show app.env app.debug app.url
php artisan config:show queue.default cache.default mail.default tenancy

Expected: app.env=production, app.debug=false, app.url starts with https://.

Common failures:

Symptom	Fix
Config cache stale	`php artisan config:clear && php artisan config:cache`
`APP_KEY` missing	`php artisan key:generate`
Still on SQLite	Set `DB_CONNECTION=pgsql`

3. Database Deployment

Why required

All CRM, finance, tenancy, and session data live in PostgreSQL. Migrations create schema; seeders bootstrap roles, plans, and optionally the first pilot clinic.

PostgreSQL setup

bash

sudo -u postgres psql <<'SQL'
CREATE USER dental_app WITH PASSWORD 'REPLACE_STRONG_PASSWORD';
CREATE DATABASE dental_clinic_ros OWNER dental_app;
GRANT ALL PRIVILEGES ON DATABASE dental_clinic_ros TO dental_app;
SQL

Application deploy

bash

cd /var/www/dental-clinic-ros
composer install --no-dev --optimize-autoloader
cp .env.example .env
# edit .env (Section 2)
php artisan key:generate
npm ci && npm run build

Migrations

bash

php artisan migrate --force

Expected output: All migrations run; ends with Nothing to migrate on re-run.

Common failures:

Error	Fix
`connection refused`	Check `DB_HOST`, Postgres listening, firewall
`permission denied`	Grant privileges to `dental_app`
Migration timeout	Increase `statement_timeout`; run off-peak

Seeders

Always required (roles + SaaS plans):

bash

php artisan db:seed --class=RolesAndPermissionsSeeder --force
php artisan db:seed --class=SaasPlanSeeder --force
php artisan db:seed --class=RegionSeeder --force

Customer #1 managed pilot (pre-built clinic + staff):

bash

php artisan db:seed --class=PilotProductionSeeder --force

Creates:

Asset	Detail
Clinic slug	`pilot-dental`
Tenant subdomain	`pilot-dental.YOURBASEDOMAIN.com`
Owner	`owner@pilot.dental` / `password` (change immediately)
Receptionist	`reception@pilot.dental` / `password`
Doctor	`doctor@pilot.dental` / `password`
Sample workflow	Lead, booking, appointment, patient, treatment, invoice, payment

Do not run PilotProductionSeeder on a live clinic database that already has real patient data.

Permissions verification

bash

php artisan tinker --execute="echo \Spatie\Permission\Models\Role::count().' roles';"
psql -U dental_app -d dental_clinic_ros -c "\dt" | head -20

Expected: Roles exist; core tables (clinics, users, leads, invoices) present.

Rollback

bash

php artisan down
# Restore DB from backup (Section 12) — do NOT rollback migrations on production without backup
php artisan up

4. Queue Infrastructure

Why required

Reminders (DispatchReminderJob), invoice emails (SendInvoiceNotificationJob), and signup notifications are queued. QUEUE_CONNECTION=sync in production fails validation and risks request timeouts.

Files

File	Setting
`.env`	`QUEUE_CONNECTION=redis`
`config/queue.php`	Connection definitions
`/etc/supervisor/conf.d/dental-clinic-worker.conf`	Worker process

Supervisor config

/etc/supervisor/conf.d/dental-clinic-worker.conf:

ini

[program:dental-clinic-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/dental-clinic-ros/artisan queue:work redis --sleep=3 --tries=3 --max-time=3600 --queue=default,ai
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
user=www-data
numprocs=2
redirect_stderr=true
stdout_logfile=/var/www/dental-clinic-ros/storage/logs/worker.log
stopwaitsecs=3600

Linux commands

bash

sudo supervisorctl reread
sudo supervisorctl update
sudo supervisorctl start dental-clinic-worker:*
sudo supervisorctl status

Laravel commands

bash

php artisan queue:work redis --once          # single job test
php artisan queue:monitor redis:default,redis:ai
php artisan queue:failed                     # list failures
php artisan queue:retry all                  # retry failed (after fix)

Verification

bash

# Dispatch test job
php artisan tinker --execute="dispatch(function(){logger('queue_ok');});"
tail -n 20 storage/logs/worker.log
redis-cli LLEN queues:default

Expected: supervisorctl status shows RUNNING; worker log shows job processed; queue length returns to 0.

Restart strategy (deployments)

bash

php artisan queue:restart
sudo supervisorctl restart dental-clinic-worker:*

Workers finish current job then exit; Supervisor restarts them.

Common failures

Symptom	Cause	Fix
Jobs table growing	Worker down	Start Supervisor
`RedisException`	Redis down / wrong host	`systemctl status redis`
Validation FAIL queue	`sync` driver	Set `QUEUE_CONNECTION=redis`
Failed jobs pile up	SMTP/API errors	`queue:failed` → fix root cause → `queue:retry`

Rollback

Stop workers during maintenance: sudo supervisorctl stop dental-clinic-worker:*. Jobs remain in Redis/DB until workers restart.

5. Scheduler

Why required

reminders:process runs every 15 minutes (bootstrap/app.php). Without cron, appointment reminders never dispatch.

Cron entry

bash

sudo crontab -u www-data -e

Add:

cron

* * * * * cd /var/www/dental-clinic-ros && php artisan schedule:run >> /dev/null 2>&1

Verification

bash

php artisan schedule:list
php artisan reminders:process

Expected schedule:list includes (among others):

*/15 * * * *  php artisan reminders:process
0    * * * *  php artisan autopilot:run          # if CLINIC_AUTOPILOT_ENABLED=true
*/10 * * * *  php artisan clinic-ai:run          # if CLINIC_AI_MANAGER_ENABLED=true
30   2 * * *  php artisan saas:billing
*    * * * *  php artisan workflow:process-schedules
0    2 * * *  php artisan backup:run              # if BACKUP_ENABLED=true

Expected reminders:process: exits 0; may output queued count.

Common failures

Symptom	Fix
`schedule:list` Redis error	Start Redis first (production cache default)
Validation says not scheduled	Ensure cron runs as `www-data`; re-run `schedule:list`
Reminders never send	Cron missing + queue worker down

Automation note (Customer #1)

With recommended pilot flags (CLINIC_AUTOPILOT_ENABLED=false, CLINIC_AI_MANAGER_ENABLED=false, CLINIC_ORCHESTRATION_EXECUTE=false), autopilot/AI manager/orchestration execute paths are disabled but reminders:process and backup:run still run.

6. Mail

Why required

SAAS_REQUIRE_EMAIL_VERIFICATION=true (default). Signup, invoice notifications, and email reminders all depend on SMTP + queue worker.

Files

File	Role
`.env`	`MAIL_*`
`config/mail.php`	Mailer config
`app/Modules/CRM/Application/Jobs/SendInvoiceNotificationJob.php`	Invoice emails
`app/Modules/CRM/Application/Services/ReminderDispatchService.php`	Reminder delivery

Test procedure

1. SMTP connectivity (swaks or artisan):

bash

php artisan tinker --execute="
\Illuminate\Support\Facades\Mail::raw('SMTP test', fn(\$m) => \$m->to('you@company.com')->subject('Pilot SMTP'));
echo 'sent';
"

2. Verification email (signup path):

Visit https://account.YOURBASEDOMAIN.com/signup
Complete signup → check inbox for verification mail

3. Reminder email:

Create appointment 25 hours ahead in admin
Ensure reminder settings enabled: Filament → CRM → Reminder Settings
Run: php artisan reminders:process
Confirm queue worker delivers email

4. Invoice email:

Issue invoice from treatment/appointment
Check jobs table / worker log for SendInvoiceNotificationJob
Confirm patient/staff notification received (if configured)

Verification

bash

php artisan pilot:validate-deployment | grep -E 'Mail|Email verification|Notification'

Expected:

[PASS] Mail (SMTP) — Mail driver [smtp] with from [...]
[PASS] Email verification — Email verification is required and mail delivery is configured.
[PASS] Notification delivery — Default reminder method [email] with mail driver [smtp].

Common failures

Symptom	Fix
Mail in log only	`MAIL_MAILER=log` → change to `smtp`
Verification blocked	Fix SMTP before onboarding
Reminder queued not sent	Start queue worker
SPF/DKIM reject	Configure DNS with mail provider

7. Tenancy

Why required

Public routes resolve clinic via tenant host (ResolvedClinicId::requireForPublicSite()). Wrong DNS → public site 500; admin may 403.

DNS records (example)

Replace YOURBASEDOMAIN.com and clinic slug pilot-dental:

Type	Name	Value
A	`account.YOURBASEDOMAIN.com`	Server IP
A	`pilot-dental.YOURBASEDOMAIN.com`	Server IP
A	`*.YOURBASEDOMAIN.com`	Server IP (wildcard tenants) OR per-clinic A records
CNAME	`crm.pilot-clinic.com`	`pilot-dental.YOURBASEDOMAIN.com` (optional custom domain)

Environment

dotenv

TENANCY_ENABLED=true
TENANCY_BASE_DOMAIN=YOURBASEDOMAIN.com
TENANCY_ENFORCE_USER_CLINIC=true
TENANCY_PUBLIC_DEFAULT_CLINIC_FALLBACK=false

Session cookie domain is set to .{TENANCY_BASE_DOMAIN} by SaasServiceProvider so login on account.* works on tenant subdomains.

Custom domain (optional)

After tenant exists:

bash

php artisan tinker --execute="
app(\App\Modules\Saas\Application\Services\TenantDomainService::class)
  ->updateCustomDomain(CLINIC_ID, 'crm.customerclinic.com');
"

Or via Filament SaaS domain settings (see PilotDeploymentReadinessTest).

Public routes to verify

URL	Expected
`https://pilot-dental.YOURBASEDOMAIN.com/en`	Homepage 200
`https://pilot-dental.YOURBASEDOMAIN.com/en/booking`	Booking form 200
`https://pilot-dental.YOURBASEDOMAIN.com/en/contact`	Contact form 200
`https://pilot-dental.YOURBASEDOMAIN.com/en/services`	Service catalog 200
`https://account.YOURBASEDOMAIN.com/admin/login`	Admin login 200

Verification

bash

curl -fsS -o /dev/null -w "%{http_code}" https://pilot-dental.YOURBASEDOMAIN.com/en
curl -fsS -o /dev/null -w "%{http_code}" https://pilot-dental.YOURBASEDOMAIN.com/up
php artisan pilot:validate-deployment | grep Tenancy

Expected: HTTP 200; Tenancy PASS with base domain shown.

Common failures

Symptom	Fix
`Public clinic context could not be resolved`	DNS not pointing; wrong subdomain in `saas_tenants`
Login works on account but not tenant	Check session cookie domain / HTTPS
403 after login	User `clinic_id` must match tenant

8. Security

HTTPS

Terminate TLS at Nginx; forward X-Forwarded-Proto: https
Set APP_URL=https://account.YOURBASEDOMAIN.com
SESSION_SECURE_COOKIE auto-enables in production (config/session.php L172)

Cookies

Default: http_only=true, same_site=lax, secure=true in production.

APP_DEBUG

dotenv

APP_DEBUG=false

Verify: trigger a Filament error → user sees generic message, not stack trace (ProductionHardeningSecurityTest).

Backups

bash

php artisan backup:run
ls -la storage/app/backups/   # or configured BACKUP_PATH

Configure off-site: BACKUP_HANDLERS in .env (see config/backup.php).

Scheduled: backup:run daily at BACKUP_SCHEDULE_TIME via bootstrap/app.php.

Storage permissions

bash

sudo chown -R www-data:www-data storage bootstrap/cache
sudo chmod -R ug+rwx storage bootstrap/cache
php artisan storage:link

Nginx (minimal PHP-FPM)

nginx

server {
    listen 443 ssl http2;
    server_name account.YOURBASEDOMAIN.com pilot-dental.YOURBASEDOMAIN.com;
    root /var/www/dental-clinic-ros/public;

    ssl_certificate     /etc/letsencrypt/live/YOURBASEDOMAIN.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/YOURBASEDOMAIN.com/privkey.pem;

    add_header X-Frame-Options "SAMEORIGIN";
    add_header X-Content-Type-Options "nosniff";

    index index.php;
    location / {
        try_files $uri $uri/ /index.php?$query_string;
    }
    location ~ \.php$ {
        fastcgi_pass unix:/run/php/php8.3-fpm.sock;
        fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name;
        include fastcgi_params;
        fastcgi_param HTTPS on;
    }
    location ~ /\.(?!well-known).* { deny all; }
}

bash

sudo nginx -t && sudo systemctl reload nginx

Verification

bash

curl -fsS https://account.YOURBASEDOMAIN.com/up
php artisan pilot:validate-deployment | grep -E 'HTTPS|Application URL'

9. Production Validation

Command

bash

cd /var/www/dental-clinic-ros
php artisan pilot:validate-deployment --strict

PASS means: Exit code 0; no FAIL lines; in --strict mode, no unreviewed WARN in production.

Check reference

Key	Label	PASS means	FAIL fix
`database`	Database	PDO connects; driver is `pgsql` in production	Install Postgres; fix `DB_*`
`app_url`	Application URL	`APP_URL` is `https://...`	Set HTTPS URL
`https_trust`	HTTPS / proxies	APP_URL uses HTTPS	TLS + proxy headers
`mail_configuration`	Mail (SMTP)	Real mailer + `MAIL_FROM_ADDRESS`	Configure SMTP
`email_verification`	Email verification	Verification on + mail ready	Fix mail or disable only if managed onboarding
`queue_worker`	Queue worker	Not `sync`	`QUEUE_CONNECTION=redis` + Supervisor
`cache_store`	Cache	Not `array`/`null`	`CACHE_STORE=redis`
`redis`	Redis	Reachable when required	`systemctl start redis`
`filesystem`	Filesystem	Disk configured	Check `FILESYSTEM_DISK`
`storage_writable`	Storage writable	Probe write/delete OK	Fix `storage/` permissions
`session_driver`	Session	Not `array`	`SESSION_DRIVER=database`
`scheduler`	Scheduler	Artisan schedule available	Laravel install intact
`schedule_reminders_process`	Scheduled: Reminder processor	In `schedule:list`	Fix cron; ensure Redis up for `schedule:list`
`schedule_autopilot_run`	Scheduled: Autopilot	In `schedule:list`	Same
`schedule_clinic_ai_run`	Scheduled: AI manager	In `schedule:list`	Same
`schedule_saas_billing`	Scheduled: SaaS billing	In `schedule:list`	Same
`schedule_workflow_process_schedules`	Scheduled: Workflow	In `schedule:list`	Same
`reminders_process`	Reminder command registered	Command exists	Deploy complete codebase
`autopilot_run`	Autopilot registered	Command exists	Same
`clinic_ai_run`	AI manager registered	Command exists	Same
`clinic_autopilot_enabled`	Autopilot enabled	Config flag (WARN if off)	Set if desired
`clinic_ai_manager_enabled`	AI manager enabled	Config flag (WARN if off)	Set if desired
`tenancy`	Tenancy	Enabled + base domain set	`TENANCY_*`
`billing_provider`	Billing provider	Provider configured (WARN if not stripe in prod)	`SAAS_BILLING_PROVIDER=stripe`
`notification_delivery`	Notification delivery	Email reminders + real mailer	SMTP + queue

Automated test suite (on staging mirror)

bash

composer test:first-customer-pack
php artisan test --filter=ProductionSmokeTest
php artisan test --filter=PilotDeploymentReadinessTest

10. Manual Smoke Test (Customer #1)

Execute on production host with real DNS. Record pass/fail per step.

Prerequisites

PilotProductionSeeder run or real clinic onboarded via signup
Staff passwords changed from defaults
Queue worker + cron running

#	Step	Action	Expected	Fail criteria
1	Health	`curl -f https://account.YOURBASEDOMAIN.com/up`	HTTP 200	Non-200
2	Homepage	Open `https://{tenant}.YOURBASEDOMAIN.com/en`	Clinic branding, no 500	404/500
3	Admin login	`https://account.YOURBASEDOMAIN.com/admin/login`	Login form loads	500/redirect loop
4	Login	Owner credentials	Dashboard loads	403/401
5	Lead creation	Public contact OR admin lead create	Lead in `LeadResource`	No record
6	Booking	Public `/en/booking` submit	Lead + booking created	Error flash
7	Appointment	Confirm booking → appointment	Appointment on calendar	Workflow stuck
8	Treatment	Create/accept treatment plan	Plan active	Billing blocked
9	Invoice	Issue invoice from treatment	Invoice `issued`	Finance error
10	Payment	Record payment in Filament	Payment linked; balance reduced	Amount mismatch
11	Finance reports	Open P&L / Outstanding reports	Correct totals	Zero/wrong data
12	Reminder	Appointment +24h; run `reminders:process`	Reminder log + email (if SMTP)	No queue job
13	AI Copilot	Open Clinic Copilot page; ask "Show today's appointments"	Response or fallback guidance	500 error
14	Compliance	Open Compliance Violations list	Page loads	500
15	Services public	Add service in admin → view `/en/services/{slug}`	DB-backed service visible	404/hardcoded only

Sign-off: Product owner + DevOps sign smoke sheet when all critical steps (1–11) pass.

11. Monitoring (First 30 Days)

Area	What to watch	How
Queue	Depth, failed jobs	`php artisan queue:monitor redis:default`; `SELECT count(*) FROM failed_jobs;`
Cron	Scheduler running	OS cron logs; `schedule:list` Next Due times advancing
Mail	Bounces, failures	SMTP dashboard; `grep invoice_notification_job_failed storage/logs/laravel.log`
Backups	Daily success	`backup:run` log; file count in `BACKUP_PATH`
Storage	Disk usage	`df -h`; `du -sh storage/app`
HTTP errors	5xx rate	Nginx `error.log`; Laravel `storage/logs/laravel.log`
App logs	Exceptions	`tail -f storage/logs/laravel.log`
Failed jobs	Stuck work	`php artisan queue:failed` daily
Database	Connections, size	`pg_stat_activity`; `pg_database_size`
Reminders	Delivery	`grep reminder_sent storage/logs/laravel.log`
Pilot validation	Weekly regression	`php artisan pilot:validate-deployment --strict`

Alert thresholds (pilot)

Failed jobs > 10 → investigate same day
Queue depth > 100 for > 15 min → restart workers
Disk > 80% → expand volume or prune logs
Any 5xx spike on /admin or /en/booking → P1

12. Rollback Plan

When to rollback

Migration failure mid-deploy
Smoke test critical failure after go-live
Security misconfiguration discovered (APP_DEBUG=true on public host)

Procedure

bash

# 1. Maintenance mode
php artisan down --retry=60 --secret="pilot-rollback-$(date +%s)"

# 2. Code rollback
cd /var/www/dental-clinic-ros
git fetch --tags
git checkout <PREVIOUS_RELEASE_TAG>
composer install --no-dev --optimize-autoloader
npm ci && npm run build

# 3. Database rollback (ONLY if migration was destructive)
# Restore from latest backup — do NOT use migrate:rollback on production without approval
pg_restore -U dental_app -d dental_clinic_ros /path/to/backup.dump

# 4. Cache clear
php artisan optimize:clear
php artisan config:cache
php artisan route:cache
php artisan view:cache

# 5. Queue recovery
php artisan queue:restart
sudo supervisorctl restart dental-clinic-worker:*
# Retry safe jobs only: php artisan queue:retry all

# 6. Verify
php artisan pilot:validate-deployment
curl -f https://account.YOURBASEDOMAIN.com/up

# 7. Up
php artisan up

Partial rollback (config only)

bash

php artisan down
cp .env.backup .env
php artisan config:clear && php artisan config:cache
php artisan queue:restart
php artisan up

13. Final Go-Live Checklist

Server & stack

[ ] Ubuntu 22.04+ provisioned
[ ] PHP 8.2+ with required extensions installed
[ ] PostgreSQL 15+ installed and reachable
[ ] Redis installed; redis-cli ping → PONG
[ ] Nginx installed with valid SSL certificate
[ ] Supervisor installed
[ ] System cron configured for www-data

Application deploy

[ ] Code deployed to /var/www/dental-clinic-ros
[ ] composer install --no-dev completed
[ ] npm ci && npm run build completed
[ ] storage/ and bootstrap/cache/ owned by www-data
[ ] php artisan storage:link executed

Environment

[ ] APP_ENV=production
[ ] APP_DEBUG=false
[ ] APP_KEY generated
[ ] APP_URL=https://account.YOURBASEDOMAIN.com
[ ] DB_CONNECTION=pgsql with working credentials
[ ] CACHE_STORE=redis
[ ] QUEUE_CONNECTION=redis (not sync)
[ ] SESSION_DRIVER=database (or redis)
[ ] MAIL_MAILER=smtp with verified MAIL_FROM_ADDRESS
[ ] TENANCY_ENABLED=true
[ ] TENANCY_BASE_DOMAIN set to production apex
[ ] TENANCY_PUBLIC_DEFAULT_CLINIC_FALLBACK=false
[ ] SAAS_REQUIRE_EMAIL_VERIFICATION=true (or managed onboarding documented)
[ ] Pilot automation flags set (CLINIC_ORCHESTRATION_EXECUTE=false recommended)
[ ] BACKUP_ENABLED=true

Database

[ ] php artisan migrate --force completed
[ ] RolesAndPermissionsSeeder run
[ ] SaasPlanSeeder + RegionSeeder run
[ ] Customer clinic created (signup OR PilotProductionSeeder)
[ ] Default passwords changed

Infrastructure services

[ ] PostgreSQL connected
[ ] Redis connected
[ ] Queue worker running (supervisorctl status → RUNNING)
[ ] Scheduler cron active
[ ] php artisan schedule:list shows reminders:process
[ ] SMTP test email delivered
[ ] HTTPS working on account + tenant hosts

DNS & tenancy

[ ] account.YOURBASEDOMAIN.com resolves to server
[ ] Tenant subdomain resolves (e.g. pilot-dental.YOURBASEDOMAIN.com)
[ ] Public homepage returns 200
[ ] Admin login returns 200
[ ] Custom domain configured (if applicable)

Security

[ ] TLS certificate valid and auto-renew configured
[ ] APP_DEBUG=false verified
[ ] Session cookies secure (HTTPS only)
[ ] .env not web-accessible
[ ] Daily backup job scheduled
[ ] Off-site backup handler configured (recommended)

Validation & smoke

[ ] php artisan pilot:validate-deployment --strict → PASS
[ ] curl -f https://account.YOURBASEDOMAIN.com/up → 200
[ ] composer test:first-customer-pack passed on staging mirror
[ ] Manual smoke test steps 1–11 passed (Section 10)
[ ] Reminder test email or log entry confirmed
[ ] Copilot page loads (fallback acceptable)

Handoff

[ ] Staff accounts created and roles assigned
[ ] Clinic owner trained on lead → payment flow
[ ] Support contact documented for pilot period
[ ] Monitoring checklist shared (Section 11)
[ ] Rollback procedure documented and .env.backup saved

GO / NO-GO: When every checkbox above is checked, run:

bash

php artisan pilot:validate-deployment --strict && echo "CUSTOMER_1_GO"

Exit code 0 + smoke sign-off = GO for Customer #1 production use.

Complements docs/deployment/PRODUCTION_RUNBOOK.md · docs/DEPLOYMENT-CHECKLIST.md

Customer #1 — Production Go-Live Runbook ​

1. Production Server Requirements ​

Why required ​

Stack ​

PHP extensions (required) ​

Initial server packages ​

Verification ​

2. Environment Configuration ​

Why required ​

Files to configure ​

Production .env template (Customer #1) ​

Variable reference (project-used only) ​

Verification ​

3. Database Deployment ​

Why required ​

PostgreSQL setup ​

Application deploy ​

Migrations ​

Seeders ​

Permissions verification ​

Rollback ​

4. Queue Infrastructure ​

Why required ​

Files ​

Supervisor config ​

Linux commands ​

Laravel commands ​

Verification ​

Restart strategy (deployments) ​

Common failures ​

Rollback ​

5. Scheduler ​

Why required ​

Cron entry ​

Verification ​

Common failures ​

Automation note (Customer #1) ​

6. Mail ​

Why required ​

Files ​

Test procedure ​

Verification ​

Common failures ​

7. Tenancy ​

Why required ​

DNS records (example) ​

Environment ​

Custom domain (optional) ​

Public routes to verify ​

Verification ​

Common failures ​

8. Security ​

HTTPS ​

Cookies ​

APP_DEBUG ​

Backups ​

Storage permissions ​

Nginx (minimal PHP-FPM) ​

Verification ​

9. Production Validation ​

Command ​

Check reference ​

Automated test suite (on staging mirror) ​

10. Manual Smoke Test (Customer #1) ​

Prerequisites ​

11. Monitoring (First 30 Days) ​

Alert thresholds (pilot) ​

12. Rollback Plan ​

When to rollback ​

Procedure ​

Partial rollback (config only) ​

13. Final Go-Live Checklist ​

Server & stack ​

Application deploy ​

Environment ​

Database ​

Infrastructure services ​

DNS & tenancy ​

Security ​

Validation & smoke ​

Customer #1 — Production Go-Live Runbook

1. Production Server Requirements

Why required

Stack

PHP extensions (required)

Initial server packages

Verification

2. Environment Configuration

Why required

Files to configure

Production `.env` template (Customer #1)

Variable reference (project-used only)

Verification

3. Database Deployment

Why required

PostgreSQL setup

Application deploy

Migrations

Seeders

Permissions verification

Rollback

4. Queue Infrastructure

Why required

Files

Supervisor config

Linux commands

Laravel commands

Verification

Restart strategy (deployments)

Common failures

Rollback

5. Scheduler

Why required

Cron entry

Verification

Common failures

Automation note (Customer #1)

6. Mail

Why required

Files

Test procedure

Verification

Common failures

7. Tenancy

Why required

DNS records (example)

Environment

Custom domain (optional)

Public routes to verify

Verification

Common failures

8. Security

HTTPS

Cookies

APP_DEBUG

Backups

Storage permissions

Nginx (minimal PHP-FPM)

Verification

9. Production Validation

Command

Check reference

Automated test suite (on staging mirror)

10. Manual Smoke Test (Customer #1)

Prerequisites

11. Monitoring (First 30 Days)

Alert thresholds (pilot)

12. Rollback Plan

When to rollback

Procedure

Partial rollback (config only)

13. Final Go-Live Checklist

Server & stack

Application deploy

Environment

Database

Infrastructure services

DNS & tenancy

Security

Validation & smoke