You set up a Multibranch Pipeline in Jenkins scanning a GitHub repo. Initially 20 branches, pipeline creates 20 jobs. Over time, the repo has 500+ branches (some stale feature branches from 2 years ago). Jenkins job list is bloated, slow to scan, consuming resources. How do you clean up and prevent future bloat?
Implement cleanup strategy: (1) Configure branch discovery to exclude stale branches. In Multibranch Pipeline config > Branch Sources > GitHub > Behaviors, add "Discover branches" with age limit: "Exclude stale branches (days old)". Set to 90 days. (2) Add "Delete stale branch" post-build step: automatically delete Jenkins jobs for deleted Git branches. (3) Use filtering: configure naming patterns to exclude branches: `^(develop|master|main|release-.*)$` matches only important branches. (4) Implement orphaned job cleanup: periodic Jenkins job that identifies jobs with no matching Git branch, deletes them. (5) Use Jenkins Cleanup plugin: scheduled job that purges jobs matching retention policy. (6) Optimize scanning: increase scan interval to reduce frequency (e.g., scan daily instead of every 5 min). (7) Monitor job count: set up alert if Multibranch exceeds threshold (e.g., 100 jobs). (8) Use repository archive: after 180 days, delete feature branches from Git repo itself. (9) Document branch lifecycle policy: feature branches deleted after PR merge + 30 days. (10) Use Jenkins metrics to track job creation/deletion trends. Best practice: combine branch discovery filters + automatic cleanup + regular manual audits.
Follow-up: A developer's branch was accidentally deleted from git during cleanup. The Jenkins job is gone. How do you prevent this?
Your Multibranch Pipeline runs different Jenkinsfile stages based on branch. The main/master branch should deploy to production. Feature branches should run tests only. However, a developer creates a branch that accidentally matches your production filtering logic, and a malicious deployment runs. How do you prevent this?
Implement branch protection and validation: (1) Use explicit branch detection: in Jenkinsfile, check branch name and environment before deployment. Example: `if (env.BRANCH_NAME == 'main' || env.BRANCH_NAME == 'master') { deploy() } else { error("Not authorized to deploy from this branch") }`. (2) Implement branch restrictions in Git: GitHub branch protection rules prevent force pushes to main/master, require PR reviews. (3) Use Declarative Pipeline syntax: define stages per branch pattern: `when { branch 'main' }` is evaluated, not skipped. (4) Implement credential-based gating: production credentials only available to main branch builds. Use Jenkins agent restrictions. (5) Use environment isolation: production deployments use separate Jenkins agents with restricted network access. (6) Implement approval gates: production deployments require manual approval before executing. (7) Use GitOps: production changes via PR merges only, triggering deployment pipeline. (8) Add validation stage: before any deployment, verify Git commit is on protected branch. (9) Use Jenkins RBAC: restrict who can approve production deployments. (10) Implement audit logging: track all deployment attempts, alert on unauthorized branch deployments. Document branch strategy: define which branches deploy to which environment. Use branch naming conventions: `feature/*`, `bugfix/*`, `release/*` to make filtering robust.
Follow-up: How do you test your branch protection logic safely without risking production?
Your Multibranch Pipeline scans 10+ repositories, each with different Jenkinsfile conventions. Some use `Jenkinsfile`, others use `Jenkinsfile-ci`, some use `.jenkins/pipeline.groovy`. Pipeline discovery is failing for some repos. How do you standardize and ensure all repos are scanned correctly?
Implement standardized discovery: (1) Enforce consistent Jenkinsfile naming: organization policy requires all repos use `Jenkinsfile` at root. (2) In Multibranch Pipeline > Branch Sources > GitHub, configure "Script path": set to `Jenkinsfile`. (3) For legacy repos with different naming: add alias configuration or Git hook that symlinks `Jenkinsfile` to actual file. (4) Use Job DSL to programmatically create Multibranch Pipelines: define scan settings in code, version-controlled. (5) Implement configuration template: shared library provides default Jenkinsfile for repos without one. (6) Use GitHub App integration instead of webhooks: better discovery reliability and rate limiting. (7) Monitor discovery failures: Jenkins logs show why repos failed scan. Alert if failure rate exceeds threshold. (8) Implement health checks: periodic validation that all repos are discoverable. (9) Use Jenkins metrics: track job creation trends per repo, alert if repo stops creating jobs (discovery failure). (10) Document standards: runbook showing teams how to structure Jenkinsfile. For migration: create CI/CD job that updates legacy repos to use `Jenkinsfile`, commits change via Git, creates PR for manual review. Use semantic versioning: `Jenkinsfile-v1`, `Jenkinsfile-v2` if breaking changes needed.
Follow-up: A repo owner wants to use `.github/workflows/main.yml` (GitHub Actions) instead of Jenkinsfile. How do you handle multi-CI repos?
Your Multibranch Pipeline has 3 long-running branches (main, develop, staging), each with heavy tests. All branches build concurrently, consuming all Jenkins executor capacity. Lower-priority feature branches queue indefinitely. Design a prioritization strategy.
Implement queue prioritization: (1) Use Jenkins Priority Queue plugin: assign priority to jobs based on branch. Main: priority 100, develop: 50, feature: 10. (2) In Jenkinsfile, set priority via script: `properties([buildDiscarder(logRotator(numToKeepStr: '30')), pipelineTriggers([githubPush()]), priority(env.BRANCH_NAME == 'main' ? 100 : 10)])`. (3) Use job throttling: configure max concurrent builds per branch. Main: 3, develop: 2, feature: 1. (4) Implement build scheduling: long-running tests on main/develop run on schedule (e.g., 2am), not on every commit. Feature branch tests run fast sanity checks only. (5) Use agent node labels: main builds use dedicated high-capacity nodes, feature builds use general pool. (6) Implement resource quotas: main branch reserved 50% of executor capacity, rest shared. (7) Use parallel stages wisely: reduce test parallelism for feature branches, maximize for main. (8) Cache optimization: main branch tests cache aggressively, feature branches use cache. (9) Use test sharding: split tests across agents, feature branches run subset. (10) Monitor queue metrics: alert if queue depth exceeds 50 jobs or wait time > 10 min. Communicate policy: teams understand that main has priority. For urgent features: manual job trigger with high priority option.
Follow-up: A feature branch's tests fail because they rely on artifacts from main branch. How do you share artifacts safely?
Your Multibranch Pipeline infrastructure runs on-prem. A repo has branches in two Git platforms: GitHub and GitLab. You need Multibranch jobs for both. Managing separate pipelines is tedious. How do you unify discovery across multiple Git platforms?
Implement multi-platform discovery: (1) Create separate Multibranch Pipelines per platform: GitHub Multibranch + GitLab Multibranch, both point to same Jenkinsfile in respective repos. (2) Use Organization Folders: Jenkins Organization Folder plugin scans GitHub/GitLab organizations, auto-creates jobs per repo. (3) Use Job DSL to programmatically define both pipelines: single DSL script creates GitHub + GitLab Multibranch jobs. (4) Implement unified triggering: GitHub webhook -> Jenkins listener triggers GitHub Multibranch, GitLab webhook triggers GitLab Multibranch separately. (5) Use GitOps bridge: central Git repo (e.g., GitHub) contains Jenkinsfile, other platforms' repos reference it as submodule. (6) For cross-platform: use Jenkins Remote Loader plugin to fetch Jenkinsfile from neutral location. (7) Implement platform-agnostic discovery: generic webhook endpoint that routes to appropriate Multibranch based on platform. (8) Use Jenkins Configuration as Code: define all Multibranch jobs in YAML, codify platform-specific settings. (9) Monitor webhook health: alert if GitLab webhook hasn't fired in 24h. (10) Document multi-platform workflow: teams understand which platform is "source of truth" for Jenkinsfile. For repos: keep Jenkinsfile identical across platforms. Use version-control sync: if Jenkinsfile changes on GitHub, sync to GitLab automatically.
Follow-up: You need to test a change to Jenkinsfile before merging. How do safely test without affecting production builds?
Your Multibranch Pipeline configuration drifts over time. Different branches have different settings (scan frequency, retention policy, timeout). Managing per-branch config is unsustainable. Implement configuration-as-code for Multibranch consistency.
Implement IaC for Multibranch configuration: (1) Use Jenkins Configuration as Code (JCasC): define Multibranch Pipeline config in YAML, version-controlled in Git. (2) Use Job DSL: create seed job that generates all Multibranch Pipelines from DSL code. (3) Store DSL in Git: any change triggers seed job, updates all Multibranch configs. (4) Define configuration template: retention policy (keep 30 builds), timeouts (60 min), scan frequency (every 5 min). (5) Override per-environment: main branch: retention 90 builds, feature branches: 10 builds. (6) Use environment variables: DSL injects values like `BRANCH_TIMEOUT`, `RETENTION_DAYS`. (7) Implement validation: lint Job DSL before applying, ensure schema compliance. (8) Use branch configuration blocks in Jenkinsfile: different stages per branch define expected config. (9) Audit changes: Git history shows who changed Multibranch config and when. (10) Implement rollback: revert Git commit to restore previous Multibranch config. Example DSL: `multibranchPipelineJob('my-app') { triggers { githubPush() } properties { buildDiscarder(logRotator(numToKeepStr: '30')) } }`. For large estates: use Terraform or Pulumi to manage Jenkins infrastructure including Multibranch jobs.
Follow-up: You need to change timeout for all Multibranch jobs across 50 repos. How do you do this safely?
Your Multibranch Pipeline uses webhooks to trigger builds on Git push. During a deployment window, hundreds of pushes occur simultaneously. Jenkins queue explodes, many builds fail due to executor exhaustion. Design a resilient webhook ingestion strategy.
Implement robust webhook handling: (1) Use message queue: webhook sends event to queue (Kafka, RabbitMQ), consumer triggers Jenkins builds. Decouples spike from Jenkins. (2) Implement rate limiting: webhook handler allows max N builds/sec, queues excess for later. (3) Use build coalescing: multiple pushes to same branch within 10 sec are coalesced into one build. (4) Implement exponential backoff: if Jenkins queue full, webhook retries with increasing delay. (5) Use pull-based polling instead of webhooks during high-traffic windows: scan repo periodically instead of event-driven. (6) Implement webhook buffering: Jenkins webhook plugin buffers incoming webhooks, processes at steady rate. (7) Use job throttling: max concurrent builds per branch to prevent executor exhaustion. (8) Implement graceful degradation: if executor queue grows, deprioritize feature branch builds, maintain main branch throughput. (9) Monitor webhook latency: alert if webhook processing delay > 1 min. (10) Use separate executor pools: webhook events use dedicated executor subset (e.g., 20 out of 100 executors), preventing starvation. Implement webhook diagnostics: Jenkins log shows queue depth, processing time, retry count. Use Splunk/ELK to analyze webhook patterns and identify optimization opportunities.
Follow-up: A webhook sender's IP address changes. Old webhooks fail. How do you handle IP rotation?
Your Multibranch Pipeline scans a large monorepo (500K files, 1000+ branches). Each scan takes 30 min, consuming heavy resources. Developers complain about latency in seeing builds start after push. Optimize scan performance.
Optimize Multibranch scanning: (1) Reduce scan frequency: increase interval from 5 min to hourly for branches that don't change frequently. Use periodic trigger instead of constant polling. (2) Use path-based filtering: configure "Discover branches" to exclude paths that don't match Jenkinsfile location. (3) Use sparse checkout: Git clone only relevant directories for monorepo, not entire tree. (4) Implement caching: Jenkins caches Git metadata between scans, reduces Git operations. (5) Use GitHub GraphQL API instead of Git API: faster branch discovery. (6) Parallelize scans: spin up multiple scan agents to scan branches concurrently. (7) Lazy evaluation: scan branches only if they contain changes (use branch protection webhooks to notify of changes). (8) Use Git shallow clone: `--depth 1` reduces clone size. (9) Implement incremental scanning: scan only new/modified branches, cache old ones. (10) Use branch exclusion patterns: ignore old/archived branches. Example configuration: exclude `^(archive|old-.*|release-20[0-1][0-9])$`. For large monorepos: consider Multirepo approach - split into smaller repos per team. Benchmark scan time before/after optimizations. Use Jenkins metrics to track scan duration trends.
Follow-up: Monorepo has 50+ services. Each service has different build requirements. How do you handle multi-service monorepo efficiently?