Building a Complete CI/CD Pipeline with GitHub Actions

What is CI/CD and Why GitHub Actions?

Continuous Integration and Continuous Deployment (CI/CD) automates the process of testing, building, and deploying your code every time you push changes. Instead of manually running tests, building Docker images, and deploying via SSH, a CI/CD pipeline does it all automatically in minutes. GitHub Actions is the most popular CI/CD platform for open-source and commercial projects because it is natively integrated with GitHub, free for public repositories, and incredibly flexible.

Core GitHub Actions Concepts

• Workflow: An automated process defined in a YAML file under .github/workflows/
• Job: A set of steps that execute on the same runner. Jobs run in parallel by default.
• Step: An individual task within a job — either a shell command or an action
• Runner: The server that runs your workflow (GitHub-hosted or self-hosted)
• Action: A reusable unit of code (e.g., actions/checkout@v4)
• Matrix builds: Run the same job with different configurations (Node versions, OS)

Complete Node.js CI/CD Pipeline

This pipeline lints the code, runs tests, builds the project, and deploys to production. It covers the full lifecycle from push to production deployment.

# .github/workflows/ci-cd.yml
name: CI/CD Pipeline

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]

env:
  NODE_VERSION: '20'
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}

jobs:
  # ===== Job 1: Lint and Type Check =====
  lint:
    name: Lint & Type Check
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: ${{ env.NODE_VERSION }}
          cache: 'npm'

      - name: Install dependencies
        run: npm ci

      - name: Run ESLint
        run: npm run lint

      - name: Run TypeScript check
        run: npx tsc --noEmit

  # ===== Job 2: Tests with Matrix =====
  test:
    name: Test (Node ${{ matrix.node-version }})
    runs-on: ubuntu-latest
    strategy:
      matrix:
        node-version: [18, 20, 22]
    services:
      postgres:
        image: postgres:16
        env:
          POSTGRES_USER: test
          POSTGRES_PASSWORD: test
          POSTGRES_DB: testdb
        ports:
          - 5432:5432
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
      redis:
        image: redis:7
        ports:
          - 6379:6379

    steps:
      - uses: actions/checkout@v4

      - name: Setup Node.js ${{ matrix.node-version }}
        uses: actions/setup-node@v4
        with:
          node-version: ${{ matrix.node-version }}
          cache: 'npm'

      - name: Install dependencies
        run: npm ci

      - name: Run database migrations
        run: npm run db:migrate
        env:
          DATABASE_URL: postgres://test:test@localhost:5432/testdb

      - name: Run tests with coverage
        run: npm run test:coverage
        env:
          DATABASE_URL: postgres://test:test@localhost:5432/testdb
          REDIS_URL: redis://localhost:6379

      - name: Upload coverage report
        if: matrix.node-version == 20
        uses: actions/upload-artifact@v4
        with:
          name: coverage-report
          path: coverage/

  # ===== Job 3: Build =====
  build:
    name: Build
    needs: [lint, test]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: ${{ env.NODE_VERSION }}
          cache: 'npm'

      - name: Install dependencies
        run: npm ci

      - name: Build application
        run: npm run build

      - name: Upload build artifact
        uses: actions/upload-artifact@v4
        with:
          name: build-output
          path: dist/
          retention-days: 7

  # ===== Job 4: Deploy =====
  deploy:
    name: Deploy to Production
    needs: build
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main' && github.event_name == 'push'
    environment:
      name: production
      url: https://myapp.com
    steps:
      - uses: actions/checkout@v4

      - name: Download build artifact
        uses: actions/download-artifact@v4
        with:
          name: build-output
          path: dist/

      - name: Deploy to Vercel
        uses: amondnet/vercel-action@v25
        with:
          vercel-token: ${{ secrets.VERCEL_TOKEN }}
          vercel-org-id: ${{ secrets.VERCEL_ORG_ID }}
          vercel-project-id: ${{ secrets.VERCEL_PROJECT_ID }}
          vercel-args: '--prod'

Docker Build and Push to Registries

# .github/workflows/docker.yml
name: Docker Build & Push

on:
  push:
    tags: ['v*.*.*']

jobs:
  docker:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write

    steps:
      - uses: actions/checkout@v4

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Login to Docker Hub
        uses: docker/login-action@v3
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}

      - name: Login to GitHub Container Registry
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Extract metadata
        id: meta
        uses: docker/metadata-action@v5
        with:
          images: |
            ${{ secrets.DOCKERHUB_USERNAME }}/myapp
            ghcr.io/${{ github.repository }}
          tags: |
            type=semver,pattern={{version}}
            type=semver,pattern={{major}}.{{minor}}
            type=sha

      - name: Build and push
        uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
          cache-from: type=gha
          cache-to: type=gha,mode=max
          platforms: linux/amd64,linux/arm64

Secrets Management

# Set secrets via GitHub CLI
gh secret set VERCEL_TOKEN --body "your-token-here"
gh secret set DATABASE_URL --body "postgres://user:pass@host:5432/db"
gh secret set AWS_ACCESS_KEY_ID --body "AKIA..."
gh secret set AWS_SECRET_ACCESS_KEY --body "wJalrX..."

# List secrets
gh secret list

# Set environment-specific secrets
gh secret set API_KEY --env production --body "prod-key"
gh secret set API_KEY --env staging --body "staging-key"

Using Secrets in Workflows

# Secrets are accessed via ${{ secrets.SECRET_NAME }}
# They are masked in logs automatically
steps:
  - name: Deploy
    env:
      API_KEY: ${{ secrets.API_KEY }}
      DATABASE_URL: ${{ secrets.DATABASE_URL }}
    run: |
      echo "Deploying with API key..."
      # The actual value is never printed in logs

Environment Protection Rules

# Define environments in your workflow
deploy-staging:
  environment:
    name: staging
    url: https://staging.myapp.com
  # No protection — deploys automatically

deploy-production:
  environment:
    name: production
    url: https://myapp.com
  # Configure in GitHub Settings > Environments:
  # - Required reviewers (1-6 people)
  # - Wait timer (e.g., 15 minutes)
  # - Deployment branches (only main)

Caching for Faster Builds

# Cache node_modules
- name: Setup Node.js
  uses: actions/setup-node@v4
  with:
    node-version: '20'
    cache: 'npm'    # Automatically caches ~/.npm

# Custom caching for other tools
- name: Cache Cypress binary
  uses: actions/cache@v4
  with:
    path: ~/.cache/Cypress
    key: cypress-${{ runner.os }}-${{ hashFiles('package-lock.json') }}
    restore-keys: |
      cypress-${{ runner.os }}-

# Docker layer caching (in docker build jobs)
- name: Build with cache
  uses: docker/build-push-action@v5
  with:
    cache-from: type=gha
    cache-to: type=gha,mode=max

Parallel Jobs and Conditional Execution

jobs:
  lint:
    runs-on: ubuntu-latest
    steps: [...]

  test-unit:
    runs-on: ubuntu-latest
    steps: [...]

  test-e2e:
    runs-on: ubuntu-latest
    steps: [...]

  # This job runs only after lint + both test jobs pass
  deploy:
    needs: [lint, test-unit, test-e2e]
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    steps: [...]

  # Conditional: run only when specific files change
  docs-deploy:
    runs-on: ubuntu-latest
    if: contains(github.event.head_commit.message, '[docs]')
    steps: [...]

Reusable Workflows

# .github/workflows/reusable-deploy.yml
name: Reusable Deploy

on:
  workflow_call:
    inputs:
      environment:
        required: true
        type: string
      url:
        required: true
        type: string
    secrets:
      deploy-token:
        required: true

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment:
      name: ${{ inputs.environment }}
      url: ${{ inputs.url }}
    steps:
      - uses: actions/checkout@v4
      - name: Deploy
        run: ./scripts/deploy.sh ${{ inputs.environment }}
        env:
          TOKEN: ${{ secrets.deploy-token }}

# .github/workflows/main.yml — calling the reusable workflow
name: Main Pipeline

on:
  push:
    branches: [main]

jobs:
  deploy-staging:
    uses: ./.github/workflows/reusable-deploy.yml
    with:
      environment: staging
      url: https://staging.myapp.com
    secrets:
      deploy-token: ${{ secrets.STAGING_TOKEN }}

  deploy-production:
    needs: deploy-staging
    uses: ./.github/workflows/reusable-deploy.yml
    with:
      environment: production
      url: https://myapp.com
    secrets:
      deploy-token: ${{ secrets.PRODUCTION_TOKEN }}

Self-Hosted Runners Setup

# On your server (Ubuntu):

# 1. Download the runner
mkdir actions-runner && cd actions-runner
curl -o actions-runner-linux-x64.tar.gz -L 
  https://github.com/actions/runner/releases/download/v2.311.0/actions-runner-linux-x64-2.311.0.tar.gz
tar xzf actions-runner-linux-x64.tar.gz

# 2. Configure (get token from GitHub > Settings > Actions > Runners)
./config.sh --url https://github.com/YOUR_ORG/YOUR_REPO 
  --token YOUR_TOKEN 
  --labels self-hosted,linux,x64

# 3. Install as a service
sudo ./svc.sh install
sudo ./svc.sh start

# 4. Use in workflow
# runs-on: self-hosted

Deploy to Kubernetes with kubectl

# .github/workflows/deploy-k8s.yml
deploy-k8s:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4

    - name: Configure kubectl
      uses: azure/setup-kubectl@v3

    - name: Set Kubernetes context
      uses: azure/k8s-set-context@v3
      with:
        kubeconfig: ${{ secrets.KUBE_CONFIG }}

    - name: Deploy to cluster
      run: |
        kubectl set image deployment/myapp 
          myapp=${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }} 
          -n production
        kubectl rollout status deployment/myapp -n production --timeout=300s

Deploy to AWS S3 + CloudFront

deploy-s3:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4

    - name: Build static site
      run: npm ci && npm run build

    - name: Configure AWS credentials
      uses: aws-actions/configure-aws-credentials@v4
      with:
        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        aws-region: us-east-1

    - name: Sync to S3
      run: aws s3 sync dist/ s3://${{ secrets.S3_BUCKET }} --delete

    - name: Invalidate CloudFront cache
      run: |
        aws cloudfront create-invalidation 
          --distribution-id ${{ secrets.CF_DISTRIBUTION_ID }} 
          --paths "/*"

Deploy via SSH

deploy-ssh:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4

    - name: Deploy via SSH
      uses: appleboy/ssh-action@v1
      with:
        host: ${{ secrets.SSH_HOST }}
        username: ${{ secrets.SSH_USERNAME }}
        key: ${{ secrets.SSH_PRIVATE_KEY }}
        script: |
          cd /var/www/myapp
          git pull origin main
          npm ci --production
          npm run build
          pm2 reload ecosystem.config.js --env production

Slack and Discord Notifications

# Notify on failure
notify:
  needs: [deploy]
  if: failure()
  runs-on: ubuntu-latest
  steps:
    - name: Slack Notification
      uses: 8398a7/action-slack@v3
      with:
        status: failure
        fields: repo,message,commit,author,action,eventName,workflow
      env:
        SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK }}

    - name: Discord Notification
      uses: sarisia/actions-status-discord@v1
      with:
        webhook: ${{ secrets.DISCORD_WEBHOOK }}
        status: failure
        title: "Deployment Failed"
        description: "Commit ${{ github.sha }} by ${{ github.actor }} failed"

Scheduled Workflows (Cron)

# Run daily at midnight UTC
on:
  schedule:
    - cron: '0 0 * * *'

jobs:
  nightly-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npm run test:e2e

  dependency-update:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npx npm-check-updates -u
      - name: Create PR if changes
        uses: peter-evans/create-pull-request@v6
        with:
          title: "chore: update dependencies"
          branch: deps/auto-update

Monorepo CI with Path Filters

on:
  push:
    branches: [main]

jobs:
  changes:
    runs-on: ubuntu-latest
    outputs:
      frontend: ${{ steps.filter.outputs.frontend }}
      backend: ${{ steps.filter.outputs.backend }}
    steps:
      - uses: actions/checkout@v4
      - uses: dorny/paths-filter@v3
        id: filter
        with:
          filters: |
            frontend:
              - 'packages/frontend/**'
            backend:
              - 'packages/backend/**'

  deploy-frontend:
    needs: changes
    if: needs.changes.outputs.frontend == 'true'
    runs-on: ubuntu-latest
    steps:
      - run: echo "Deploying frontend..."

  deploy-backend:
    needs: changes
    if: needs.changes.outputs.backend == 'true'
    runs-on: ubuntu-latest
    steps:
      - run: echo "Deploying backend..."

Local Testing with act

# Install act (run GitHub Actions locally)
brew install act         # macOS
choco install act-cli    # Windows

# Run all workflows
act

# Run a specific workflow
act -W .github/workflows/ci-cd.yml

# Run a specific job
act -j test

# List all workflows
act -l

# Use a specific event
act pull_request

# Pass secrets
act -s GITHUB_TOKEN=your_token
act --secret-file .secrets

Troubleshooting Common Errors

Problem: "Resource not accessible by integration"

Cause: The workflow lacks necessary permissions.

Solution:

permissions:
  contents: read
  packages: write
  pull-requests: write

Problem: Cache miss every time

Cause: The cache key changes on every run.

Solution: Use a hash of your lock file.

key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}

Problem: "Error: Process completed with exit code 1"

Cause: A command failed. Check the step logs for the real error.

Solution:

# Add debugging
- name: Debug step
  run: |
    set -x  # Print all commands
    npm run build 2>&1 | tee build.log
  env:
    CI: true

Quick Reference Cheat Sheet