Reliability questions are where senior DE candidates separate from mid-level ones. It's not about knowing Airflow syntax — it's about what you do when things break at 2am.
This guide covers the reliability topics that appear most in senior DE interviews, what interviewers are actually listening for, and how to structure answers that demonstrate production experience.
Why Reliability Is a Senior Signal
Interviewers ask reliability questions for a specific reason: they're expensive to fake.
You can study SQL and recite CAP theorem definitions. But knowing how to handle a silent data quality failure in production, or designing a retry strategy that doesn't amplify load on a failing upstream API — that only comes from having built and broken real systems.
When they ask "what happens when your pipeline fails?", they're not looking for "I'd add error handling." They want to hear you describe the real failure modes, the decision points, and what you've actually seen go wrong.
The Core Reliability Topics
1. Idempotency Design
The question: "How do you ensure your pipeline can be safely re-run?"
Idempotency means running the same operation multiple times produces the same result as running it once. It's the foundation of reliable pipelines — without it, retries and backfills corrupt your data.
What interviewers want to hear:
- The distinction between idempotent and at-least-once execution
- How you handle this at the storage layer (
INSERT OVERWRITEvsINSERT INTO) - The role of
execution_date/logical_datein Airflow idempotency - Why you should never use
datetime.now()inside a pipeline task
Strong answer pattern:
"I design for partition-level idempotency. Each pipeline run writes to a date-partitioned table using
INSERT OVERWRITE PARTITION. If it runs twice for the same partition, the second run replaces the first — so re-running after a failure is safe by default. Within the task, I only use the execution date (not wall clock time) for any time-based logic, and I validate row counts against expectations before swapping the target table."
The partition overwrite detail is the key — it shows you've actually solved this in a storage-aware way, not just added a try/catch.
2. Retry Strategy Design
The question: "How do you design retry logic for a pipeline that calls an external API?"
Naive retries can make failures worse. Interviewers want to see you think about failure modes, not just "add retries."
The concepts that matter:
Exponential backoff with jitter — don't retry every 30 seconds. Each retry waits 2× longer, plus a random jitter offset. Without jitter, all failed tasks retry simultaneously and hammer the recovering service.
import random, time
def backoff(attempt: int, base: float = 1.0, cap: float = 60.0) -> float:
return min(cap, base * (2 ** attempt)) + random.uniform(0, 1)
Distinguishing retryable vs non-retryable errors:
429 Too Many Requests→ retryable with backoff400 Bad Request→ not retryable (your payload is malformed)503 Service Unavailable→ retryable404 Not Found→ usually not retryable
Dead-letter queues / dead-letter tables: where messages/records go after exhausting retries. You need to be able to inspect, fix, and replay them — not just lose them silently.
What separates strong answers: mentioning that retries should be idempotent (see above) and that you should alert on dead-letter accumulation, not just on initial failures.
3. Incident Response
The question: "Your morning batch job is 3 hours late. Walk me through what you do."
This is a pure production scenario. There's no single right answer — they want to see your diagnostic process.
The structure that works:
Step 1: Scope before you dig. Is it this job only, or are multiple pipelines slow? Is there a cluster-wide issue (Spark, Snowflake warehouse, network)? This takes 2 minutes and saves 2 hours of wrong debugging.
Step 2: Check the obvious first.
- Did upstream data arrive? (Volume check on source)
- Is there unusual data volume? (Explain plan / row count at each stage)
- Resource contention? (Check query queue, cluster utilisation)
Step 3: Communicate early. Even before you know the cause — stakeholders who depend on the data need to know there's an incident. A brief Slack message ("Morning batch is delayed, investigating, ETA in 30 min") is always the right move.
Step 4: Have a stop-loss. At what point do you declare the run failed and move on? If the job is still running at T+4h, does it get killed? Does a downstream alert get triggered?
What trips candidates up: jumping straight to step 2 without scoping. Interviewers flag this — it's the pattern of someone who debugs alone rather than thinking about system-wide impact.
4. Silent Failures and Data Quality
The question: "How do you detect when a pipeline succeeded but produced wrong data?"
This is harder than handling crashes. The pipeline ran. Airflow shows green. The data is wrong.
The layered approach:
Layer 1 — Schema validation at ingestion. Does the incoming data match the expected schema? Column counts, types, nullability constraints. This catches upstream schema drift early.
Layer 2 — Statistical assertions in the pipeline. Not just "did rows load" but "are the aggregates in expected range?" Tools like dbt tests, Great Expectations, or custom SQL assertions:
-- Fail if daily revenue is >3 standard deviations from 90-day average
SELECT COUNT(*) FROM daily_metrics
WHERE ABS(revenue - avg_90d) > 3 * stddev_90d
Layer 3 — Cross-system reconciliation. Periodic checks that your warehouse totals match source system totals. If your DWH shows 1.2M orders and the source API reports 1.25M, something is wrong.
Layer 4 — Monitoring lag, not just failures. Even if data looks correct, are your SLOs being met? A table that usually refreshes by 8am but today refreshed at 11am is a reliability issue even if the data is right.
5. Disaster Recovery and Runbooks
The question: "How would you design the DR strategy for a critical data pipeline?"
They're testing whether you think about recovery before incidents, not during.
Key concepts:
RPO (Recovery Point Objective): How much data can you afford to lose? If RPO is 1 hour, you need backups at least every hour.
RTO (Recovery Time Objective): How long can the system be down? If RTO is 4 hours, your DR process must complete in 4 hours.
Runbooks: Step-by-step, copy-pasteable instructions for recovering specific failures. Not "restore from backup" but exactly which commands to run, in what order, with what validation steps. Written when you're not in an incident, tested regularly.
What interviewers love to hear: that you've actually tested your recovery process. "We had a quarterly DR drill where we simulated a region failure and measured actual RTO" is a strong signal.
The Pattern Behind Strong Reliability Answers
Every strong reliability answer has three parts:
- The mechanism — what specifically you do or design
- The failure mode it prevents — why this matters
- The edge case — what still doesn't work and how you handle it
Candidates who hit all three demonstrate that they've shipped production systems and debugged real failures, not just studied for interviews.
Practise these reliability scenarios with AI feedback on your answers. polydomainai covers incident response, retry design, postmortems, and disaster recovery — all with AI-scored rubric feedback.