You're building a Jenkinsfile that runs tests against 5 Python versions (3.8, 3.9, 3.10, 3.11, 3.12) and 3 databases (PostgreSQL, MySQL, MongoDB). Using sequential stages, total build time is 8 hours. Parallelize using Matrix builds without multiplying job count.
Use Declarative Pipeline Matrix build: (1) Define matrix axes in Jenkinsfile: `matrix { axes { axis { name: 'PYTHON_VERSION', values: ['3.8', '3.9', '3.10', '3.11', '3.12'] }, axis { name: 'DATABASE', values: ['postgres', 'mysql', 'mongodb'] } } }`. (2) This creates 5x3=15 parallel combinations automatically. (3) Single Multibranch job spawns all combinations as stages. (4) Each combination runs on separate executor simultaneously. (5) Example stage: `sh 'docker run -e DB=$DATABASE python:$PYTHON_VERSION python -m pytest tests/'`. (6) Implement failure resilience: use `failFast: false` to run all combinations even if one fails. (7) Collect results: use junit plugin to aggregate test results across all combinations. (8) Visualize via Blue Ocean: matrix builds show as grid. (9) Archive artifacts: each combination archives logs separately. (10) Implement exclusion: skip certain combos (e.g., MongoDB only tested on Python 3.10+): `excludedAxes { excludeAxis { combinationMatcher: 'DATABASE=="mongodb" && PYTHON_VERSION in ["3.8", "3.9"]' } }`. Expected result: build completes in ~30 min (8h / 16 parallel streams). Monitor: track executor usage to ensure cluster isn't over-utilized.
Follow-up: One Python version consistently fails on a specific database combo. How do you debug without running all 15 combinations?
Your pipeline uses Matrix builds with 20 combinations. One combination fails, blocking the entire build (failFast: true). You need to skip specific failing combinations temporarily while fixing the root cause. Implement selective skipping.
Implement conditional matrix execution: (1) Add environment-based exclusion: `excludedAxes { excludeAxis { combinationMatcher: '(PYTHON_VERSION=="3.8" && DATABASE=="mongodb")' } }` skips known-failing combo. (2) Use Jenkinsfile parameters: pass list of excludes at build time. (3) Implement dynamic exclusion: query test history, auto-exclude combos that failed >70% of recent builds. (4) Use a skip file in Git: `MATRIX_SKIP.txt` contains failing combinations, pipeline reads and excludes. (5) Implement fallback logic: if Python 3.8 fails, automatically run with Python 3.9 as substitute. (6) Use Jenkins artifacts: previous build's test results determine which combos to skip. (7) Implement regression detection: run quick sanity tests on all combos, full tests only on passing combos. (8) Use post-build action to mark failing combos: generate report of which combos failed, skip them on next build. (9) For temporary fixes: create test-only branch with exclusions, merge once root cause fixed. (10) Communicate via Slack: notify team of skipped combos, link to issue tracking fix. Example: `MATRIX_SKIP_PATTERNS='3.8.*mongodb,3.9.*mysql'` prevents those combinations from running.
Follow-up: How do you re-enable a previously skipped combination to verify the fix?
Your Matrix build spawns 50 parallel combinations. Executor pool is only 40. Builds queue up, executor capacity thrashes with job scheduling overhead. Design efficient executor allocation for matrix builds.
Optimize executor usage: (1) Implement batch execution: limit matrix concurrency to executor count. Jenkins does this natively with `maxConcurrency: 40`. (2) Use node labels: assign matrix stages to specific agent pool. Example: `agent { label 'matrix-builder && high-memory' }`. (3) Reduce matrix size: combine orthogonal axes into single variable. (4) Implement queue throttling: Jenkins-wide setting limits concurrent builds. (5) Use executor dedicated pools: reserve executors for matrix, prevent other jobs from stealing them. (6) Optimize stage duration: parallelize test execution within each combination to reduce total time. (7) Use container-based parallelism: run Kubernetes pods per combination instead of requiring dedicated Jenkins executors. (8) Implement smart scheduling: schedule matrix builds during off-peak hours if executor contention is high. (9) Use conditional parallelism: run fewer combinations on developer branches, full matrix on main branch. (10) Monitor executor utilization: Prometheus metrics show executor load; scale executor pool if consistently >80% utilization. For Kubernetes: use Kubernetes Jenkins plugin to dynamically spawn agents per matrix combination, auto-scale down when done. This eliminates executor bottleneck entirely.
Follow-up: A developer triggers 10 matrix builds simultaneously, saturating executors. How do you prevent this?
Your Matrix build has dynamic axes: number of combinations changes based on Git branch or build parameters. Some developers create branches that generate 100+ combinations, causing resource spikes. Implement safety limits.
Implement matrix safeguards: (1) Set max matrix combinations: `properties([buildDiscarder(logRotator(numToKeepStr: '30')), parameters([string(name: 'MAX_MATRIX_COMBOS', defaultValue: '50', description: 'Max matrix combinations allowed')])])`. (2) Add pre-pipeline validation: check combination count before matrix executes. If exceeds limit, fail build with error message. (3) Implement dynamic axis filtering: exclude low-priority axes on non-main branches. (4) Use input step for large builds: `input message: 'This will create 100+ combinations. Continue?', ok: 'Run'`. (5) Implement combination quotas per branch: main: 100 combos, feature: 20 combos. (6) Use branch pattern matching: `if (env.BRANCH_NAME =~ /^release\/.*/) { combos = full_list } else { combos = minimal_list }`. (7) Set timeout for matrix execution: if any combination exceeds 30 min, kill it. (8) Implement resource monitoring: fail build if matrix would consume >80% executor capacity. (9) Use Jenkins Job DSL to validate matrix config before creation. (10) Document matrix guidelines: teams understand limits and rationale. Example Groovy: `if (MATRIX_COMBOS.size() > 50) { error("Matrix too large: ${MATRIX_COMBOS.size()} combos. Max 50.") }`.
Follow-up: A legitimate use case requires 200 combinations. How do you support this safely?
You're running parallel stages in a pipeline. One stage modifies shared state (writes to artifact cache), causing race conditions. Subsequent stages read corrupt data. Implement thread-safe artifact sharing in parallel stages.
Implement safe parallel artifact handling: (1) Use separate artifact directories per stage: each parallel stage writes to `artifacts/${STAGE_ID}/` instead of shared location. (2) Use named locks: Groovy script acquires mutex before writing shared state: `synchronized(lockObj) { write_to_cache }`. (3) Use file locking: `flock` command prevents concurrent writes. Example: `flock -x /tmp/cache.lock -c 'write_operation'`. (4) Implement eventual consistency: stages write to local temp, Jenkins collects artifacts post-stage. (5) Use Jenkins lockable resources: define shared resource, stages acquire lock before use: `lock(resource: 'artifact_cache', skipIfLocked: true) { ... }`. (6) Use atomic operations: write to temp file, rename atomically (atomic on most filesystems). (7) Implement version control for artifacts: each write creates versioned artifact with timestamp, readers pick latest. (8) Use distributed caching: Redis/Memcached instead of shared filesystem. (9) Implement conflict detection: if concurrent writes detected, fail build and retry. (10) Use artifact fingerprinting: Jenkins detects when artifact changes, warns if multiple stages wrote to same artifact. Example: `archiveArtifacts artifacts: 'dist/app-${BUILD_ID}-${STAGE_NAME}.jar'` ensures no collision.
Follow-up: A parallel stage fails partway through. Other stages' artifacts are now missing. How do you recover?
Your parallel stages include both fast (2 min) and slow (30 min) stages. Pipeline waits for slowest stage before proceeding. The final stage (deployment) starts 30 min after fast stages complete. Optimize overall build time using stage result propagation.
Implement stage result management: (1) Use `waitForStage` plugin or build-in Pipeline features to start dependent stages without waiting for all parallel stages. (2) Define stage dependencies: stage B depends on stage A, not all parallel stages. (3) Implement pass-through conditions: fast stages complete, notify downstream immediately. Slow stages continue in background. (4) Use asynch execution: deploy after fast tests pass, slow tests run in parallel. (5) Implement stage-specific timeouts: fast stages timeout at 5 min, slow at 45 min. (6) Use quality gates: if fast tests fail, skip slow tests and deployment. (7) Implement asynchronous artifact delivery: fast stage produces artifact, deployment doesn't wait for slow stage. (8) Use Jenkins pipeline feedback: fast stages mark build as "ready for deployment", slow stages mark as "verified". (9) Implement conditional parallelism: if on main branch, run all tests. If feature branch, run fast tests, defer slow tests to merge. (10) Use build cache: slow tests cache results, reuse across builds if input unchanged. Strategy: parallel all tests, but don't block deployment on slow tests. Use input step with timeout: `input(message: 'Deploy?', ok: 'Deploy', submitterParameter: 'approver', timeout: time(unit: 'HOURS', 1))` allows manual override.
Follow-up: A slow test fails after deployment has started. How do you handle this?
Your Matrix build with 30 combinations produces 30 sets of test results, logs, and coverage reports. Aggregating and visualizing results across combinations is complex. Design a centralized result reporting system.
Implement centralized result aggregation: (1) Use JUnit plugin to collect all XML reports: `junit '**/test-results/*.xml'` aggregates across all combinations. (2) Use Coverage plugin: `publishCoverage adapters: [opencovAdapter('coverage-report.xml')]` merges coverage across combos. (3) Implement result parsing: post-build script aggregates results into summary table. (4) Use Jenkins metrics plugin: export build metrics (pass/fail count per combo) to Prometheus. (5) Implement result visualization: create HTML report showing pass/fail matrix grid. (6) Use Blue Ocean pipeline view: native matrix visualization shows all combinations. (7) Implement email reports: mail plugin sends summary with pass rates per combination. (8) Use webhooks for reporting: trigger external dashboard to display results. (9) Implement artifact collection: tarball all logs/reports, archive centrally. (10) Use trend analysis: track pass rates over time, alert if combination regresses. Example Groovy: aggregate test results via groovy postbuild: `def results = []; new File('test-results').eachFile { results << readJSON(file: it) }; writeJSON(file: 'summary.json', json: results)`. For Kubernetes: use centralized logging (ELK) to collect logs from all pod combinations, queryable dashboard.
Follow-up: A combination's test results are outliers (way slower/more failures than others). How do you investigate?
You're implementing a parallel stage that handles both success and failure gracefully. One stage publishes artifacts, another sends notifications, a third records metrics. If publish fails, notifications should still run but metrics recording should skip. Implement conditional post-actions for parallel stages.
Implement conditional parallel post-actions: (1) Use post block with conditions: `post { always { notifySlack() } failure { recordMetricsFailed() } success { recordMetricsSuccess() } }`. (2) Use try-catch in parallel stages: stage wraps in try-catch, catch block handles partial failure. (3) Implement stage result tracking: store result of each stage in map, subsequent stages check map. (4) Use Jenkins step result codes: `sh 'command' || true` continues pipeline on failure, captures exit code. (5) Implement conditional steps: `if (currentBuild.result != 'FAILURE') { publishArtifacts() }`. (6) Use lockable resources for serialization: if artifact publish requires exclusive access, use lock. (7) Implement retry logic: failed publishes retry before proceeding to post-actions. (8) Use Jenkins pipeline input: manual override for failed stages. (9) Implement result aggregation: post stage collects results from all parallel stages, makes decisions. (10) Use Jenkins plugins like Pipeline Aggregator to manage complex post-action logic. Example: `parallel( publish: { publishArtifacts() }, notify: { notifySlack() }, metrics: { try { recordMetrics() } catch (e) { echo "Metrics failed: $e" } } )` runs all three, doesn't block on failures.
Follow-up: You need to rollback an artifact if metrics recording detects anomaly. How do you implement this?