"Field notes, war stories, and hard-won lessons from the trenches of DevOps and beyond."
I'd been asleep for maybe two hours when my phone started screaming. Three alerts in 30 seconds — that's never just a blip. Something was genuinely wrong, and it had my name all over it.
There was a period — about three months into my first serious K8s deployment — where I genuinely believed Kubernetes was designed to break engineers. Not software. Engineers.
We had a legacy app. It worked on my machine. It worked on the staging server — most of the time, on Tuesdays, when the weather was right. But the moment we tried to containerize it, every dirty secret the codebase had been keeping came out screaming.
The product manager had said "we're expecting maybe 100 concurrent users at launch." I built exactly for that. And then launch day happened, and somewhere between 800 and 1,000 users, the whole thing fell over like a folding table at a wedding.
I used to think whiteboarding was a performance — something senior engineers did to look smart in meetings. Then I skipped it once on a "simple" feature, wrote 3,000 lines of code, and had to rewrite 2,400 of them two weeks later.
It was a Friday afternoon — never a good time for production incidents — when our payment service started timing out. Not because of the payments themselves, but because of what happened three steps after a payment succeeded.
Five years ago, I thought DevOps was mostly about writing clever bash scripts and knowing which button to press when things broke. I was wrong about almost everything except the bash scripts.
My resume says I'm proficient in Ansible, Terraform, Helm, ArgoCD, Prometheus, Grafana, and about fourteen other tools that sound very impressive at interviews. The truth is messier and much funnier.
I've been using AI tools in my DevOps workflow for over a year now. Copilot for code, ChatGPT for runbooks, Claude for incident postmortems. And I have opinions — unpopular ones in some circles.
"No scrolls found in this archive. The chronicler is still writing..."
by Adarsh Shankar · Senior DevOps Engineer
I'd been asleep for maybe two hours when my phone started screaming. Three alerts in 30 seconds — that's never just a blip. Something was genuinely wrong, and it had my name all over it because I had been the last person to touch the pipeline that day.
It was a Thursday night. I know you've heard the rule about not deploying on Fridays. We hadn't deployed on a Friday. We deployed on a Thursday evening, which is almost as bad, because by the time anyone notices something's wrong, it's already Friday.
We had a GitHub Actions pipeline set up for a microservice that handled user notifications. Nothing fancy — build, test, push to registry, deploy to production. The deploy job had been running for months without incident.
What we hadn't accounted for was a race condition between two parallel jobs. Job A was supposed to migrate the database schema. Job B was supposed to deploy the new service version. In theory, A ran before B. In practice, on this particular run, both jobs grabbed a runner at the same time.
# What we thought the pipeline looked like
job:migrate-db → job:deploy-service
# What actually happened at 11:47 PM
job:migrate-db ──→ (waiting for runner)
job:deploy-service → (got runner first) → DEPLOYED
job:migrate-db ──→ (ran second) → SCHEMA MISMATCHThe new service was live. The old schema was in place. Every write operation started throwing 500s. Users who opened the app saw a blank screen or an error. We didn't have monitoring alerts on the migration job specifically, so we found out from customer support — at 2AM.
When I finally got to my laptop, the first thing I did was the worst possible thing: I panicked and tried to redeploy the previous version. Except I didn't check which previous version. I rolled back to something from two weeks ago.
Now we had three problems instead of one: the schema mismatch, the rolled-back service version, and the fact that two weeks ago we had different environment variables in the pipeline. The app came back up but half the features were broken for different reasons.
"When production is on fire, the most expensive thing you can do is act fast without thinking. Take thirty seconds. Write down what you know. Then move."
My team lead called at 2:40AM, very calm, very measured, and very clearly suppressing the urge to ask me what I'd done to his Thursday night. We got on a call, went through the deployment logs properly, identified the root cause, and had a plan in 20 minutes. The fix took another 45.
After the postmortem, we made three concrete changes:
# Fixed pipeline: explicit dependency
jobs:
migrate-db:
runs-on: ubuntu-latest
steps:
- run: ./scripts/migrate.sh
deploy-service:
needs: migrate-db # This line. This entire line.
runs-on: ubuntu-latest
steps:
- run: ./scripts/deploy.shThe technical fix was simple — one line of YAML. But the real lesson was about assumptions. I had assumed the pipeline ran in order. I had assumed we'd know immediately if something went wrong. I had assumed rolling back would be straightforward.
Assumptions in production are just bugs you haven't found yet.
The second lesson: write your rollback procedure before you deploy, not after something breaks. If you can't describe in plain English how you'd undo a deployment in the next five minutes, you're not ready to deploy.
What's your worst 2AM incident story? And more importantly — did it change how you deploy?
by Adarsh Shankar · Senior DevOps Engineer
There was a period — about three months into my first serious K8s deployment — where I genuinely believed Kubernetes was designed to break engineers. Not software. Engineers. The software was fine. I was the one in pieces.
Pods were crashing for reasons that didn't make sense. The cluster looked healthy in the dashboard. My services weren't running. I was adding resource limits I barely understood and watching things crash faster. I called it "Kubernightmares" in my head and told everyone the technology was "overcomplicated for our scale."
I was wrong. But it took a specific incident to show me why.
We had a Python API pod that would start up, run for 40–60 seconds, and then die. The status was always OOMKilled. I kept bumping the memory limit — 256MB, 512MB, 768MB. Still dying. I was convinced Kubernetes had a bug.
# What kubectl kept telling me
$ kubectl describe pod api-pod-xyz
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
# I kept doing this and expecting different results
$ kubectl set resources deployment/api --limits=memory=768MiThe turning point was when a colleague said three words: "Check the app."
The app had a memory leak. A large list was being loaded into memory on every API request and never cleared. After 60 seconds of moderate traffic, memory usage crossed whatever limit I'd set. Kubernetes wasn't killing my pod arbitrarily. Kubernetes was telling me your app has a memory problem, and I'd been shooting the messenger.
This became my mental model shift. Kubernetes doesn't fail silently. Every crash loop, every OOMKill, every failed liveness probe is a communication. The cluster is telling you something specific. The question is whether you're listening or just trying to silence the alarm.
# Liveness probe — K8s checks if your app is alive
livenessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 30 # Give the app time to start
periodSeconds: 10
# Readiness probe — K8s checks if your app should receive traffic
readinessProbe:
httpGet:
path: /ready
port: 8080
initialDelaySeconds: 10
periodSeconds: 5Resource limits are Kubernetes saying "if your app needs more than this, something is wrong." They're not arbitrary constraints — they're a forcing function for understanding your application's actual behavior.
Before I understood this, I treated limits as obstacles. After the OOMKill incident, I started treating them as requirements — something I needed to measure, not guess. I ran load tests, watched memory usage, set limits at 1.5x the observed peak with a reasonable buffer. Pods stopped dying.
Kubernetes isn't complex. It's honest about the complexity you already had in your application. The frustration isn't with K8s — it's with discovering that your app had problems you were comfortable ignoring before containerization forced them into the open.
About six months in, I was pairing with a junior engineer who was going through the same frustration I'd had. "This thing is impossible," she said, staring at a CrashLoopBackOff. I smiled, because I'd said the exact same thing.
We spent twenty minutes reading the logs properly. Found a missing environment variable. Pod started. She looked at me and said, "That's it?" That's it. Kubernetes had been telling us the whole time.
If you're in your Kubernetes frustration phase right now — the phase where you're cursing Kelsey Hightower and whoever invented YAML — just try this: before you change any configuration, ask yourself what the error is actually saying. The answer is almost always in the logs.
What was your Kubernetes turning point? Did you have one, or are you still in the fighting phase?
by Adarsh Shankar · Senior DevOps Engineer
We had a legacy app. It worked on my machine. It worked on the staging server — most of the time, on Tuesdays, when the weather was right and the database connection pool hadn't been exhausted by someone running a test the wrong way. But the moment we tried to containerize it, every dirty secret the codebase had been keeping came out screaming.
I want to talk about what I learned, because it was genuinely embarrassing and also genuinely valuable.
The first version of the Dockerfile I wrote for this app was 40 lines long and produced a 1.8GB image. For a Python web service. I had RUN pip install -r requirements.txt after a COPY . ., which meant every time any file changed, Docker rebuilt the entire dependency layer. My build times were 12 minutes. My teammates were not happy.
# The crime scene (don't do this)
FROM python:3.11
COPY . .
RUN pip install -r requirements.txt
CMD ["python", "app.py"]
# What it should look like (multi-stage, cache-friendly)
FROM python:3.11-slim AS builder
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
FROM python:3.11-slim
WORKDIR /app
COPY --from=builder /usr/local/lib/python3.11/site-packages /usr/local/lib/python3.11/site-packages
COPY . .
CMD ["gunicorn", "app:app", "--workers", "4"]After switching to a multi-stage build and a slim base image, we got to 320MB. After adding a proper .dockerignore file, 290MB. These aren't just performance wins — a smaller image has a smaller attack surface, pulls faster in CI, and costs less to store in your registry.
Here's the thing about "works on my machine" — your machine has years of accumulated state. Packages installed by other projects. Environment variables set in ~/.bashrc that you've forgotten about. Global tools that you installed once in 2022 and never thought about again.
When we tried to containerize the legacy app, Docker's clean environment revealed all of it:
"Works on my machine" is a statement about your machine's undocumented state, not a statement about your code. Docker exposes the difference.
The .dockerignore file is the most underappreciated file in a containerized project. Without it, you're copying your .git directory, your node_modules, your .env files, and your developer's local config into the image.
# A .dockerignore worth having
.git
.gitignore
node_modules
__pycache__
*.pyc
.env
.env.local
Dockerfile
docker-compose*.yml
README.md
tests/
.pytest_cacheBy the time we had a working, clean Docker image for this legacy app, I understood it better than I had in two years of working on it. I knew every external dependency. I knew exactly what environment variables it needed. I knew which system libraries it relied on. I had documented all of this — not in a wiki that no one reads, but in the Dockerfile itself, which is just a really well-structured installation script.
If you have a service that's "hard to set up" or "only works on certain machines," containerize it. Not because you need to deploy it to Kubernetes. Just to find out what it actually needs. The exercise alone is worth it.
What was the most surprising thing your Dockerfile revealed about your app?
by Adarsh Shankar · Senior DevOps Engineer
The product manager had said "we're expecting maybe 100 concurrent users at launch." I built exactly for that. And then launch day happened, and somewhere between 800 and 1,000 users, the whole thing fell over like a folding table at a wedding — catastrophically, publicly, and with snacks everywhere.
I don't blame the PM. I blame the fact that I heard "100 users" and treated it as the ceiling of the universe, not a lower bound for planning.
Single Node.js API server. Single PostgreSQL database. No caching. No load balancer. Session state stored in the database. Every single user action hit the DB directly — even reads that could have been cached. Every session lookup was a DB query.
For 100 users, this is fine. For 1,000 concurrent users all doing session lookups and basic reads, you're making the database do the work of what should be a cache, and the database starts queuing requests, and latency climbs, and users retry, which creates more requests, which makes things worse. This is called a thundering herd, and it's not a subtle problem once it starts.
# Symptom: DB connection pool exhausted
Error: too many connections for role "api_user"
FATAL: sorry, too many clients already
# What the monitoring showed (when we finally had monitoring)
DB active connections: 487 / 100 max
API response time p99: 45,000ms
Error rate: 67%1. Connection pooling with PgBouncer. The database was never the bottleneck itself — our connection count was. PgBouncer sits between the app and the database, maintains a pool of actual DB connections, and multiplexes application connections through them. We went from 487 raw connections to 40 pooled connections serving the same load.
2. Redis for sessions and hot reads. Session lookups don't need to hit the database. We moved session state to Redis and cached any read that was requested more than three times per minute. The database query rate dropped by 70% immediately.
# Before: every session check = DB query
SELECT * FROM sessions WHERE token = $1 -- ~200 times/sec
# After: Redis first, DB only on miss
redis.get(`session:${token}`) # ~1ms
→ if miss: db.query() → redis.set() # only on cold start3. Horizontal scaling behind a load balancer. Once sessions were out of the app, we could run multiple instances. We moved to an Application Load Balancer with three API instances. Stateless services can scale horizontally. Stateful ones can't. This is the difference.
Scale isn't a feature you add when you need it. It's a foundation you build from the start — or at least, you build it knowing it will need to scale. The specific decisions that killed us were:
"Scale isn't a feature, it's a foundation. You don't add it later — you either have it or you don't when you need it."
None of these are expensive or complex to add upfront. A Redis instance costs almost nothing. PgBouncer is free. The load balancer adds maybe $20/month. The cost of not having them was a failed launch and three days of emergency engineering work.
The next system I built, the first thing I asked wasn't "what features do we need?" It was "what happens when this gets ten times the expected load?" The answer to that question changes your architecture more than any feature requirement will.
What was the scale assumption you made that turned out to be catastrophically wrong?
by Adarsh Shankar · Senior DevOps Engineer
I used to think whiteboarding was a performance — something senior engineers did to look smart in meetings. Then I skipped it once on a "simple" feature, wrote 3,000 lines of code, and had to rewrite 2,400 of them two weeks later. Now I whiteboard everything, including things that don't need it, just to prove to myself that I understand what I'm building.
Let me tell you about the feature that changed my mind.
The requirement was: "Users should be able to export their data as a CSV." Simple. I knew how to write CSV. I knew how to stream files. I started coding.
Three days later, I had a working implementation. It timed out for any user with more than 5,000 records. HTTP connections have timeouts. The export for large accounts took 4 minutes. Nobody had mentioned that we had accounts with 200,000+ records.
If I had drawn one box (User), one box (API), one box (Database), and one arrow (export all records), I would have immediately seen the problem: that arrow was doing too much work synchronously. The design would have taken 10 minutes. The rewrite took three days.
System design thinking happens at two levels, and both matter:
High-Level Design (HLD) is about components and how they talk to each other. What services exist? What databases? What queues? Where does data flow? You don't care about function names or database schemas at this level — you care about boundaries and interfaces.
Low-Level Design (LLD) is about what's inside each component. Schema design, API contracts, class structures, algorithm choices. This is where you get into specifics.
Most junior engineers skip straight to LLD because it feels more like "real" work. Most production incidents happen because the HLD was wrong.
The CSV export problem was an HLD problem. The right design was:
Three boxes, four arrows. Ten minutes on a whiteboard. Would have saved 60 hours of engineering time across two engineers.
The answer isn't that senior engineers are smarter or more disciplined. It's that they've been burned enough times to know that code is expensive to change and drawings are free.
A box you erase and redraw takes 5 seconds. A service you refactor takes 2 days and a PR review. Senior engineers have an intuitive sense for the cost of changing their mind at different stages of development, and they front-load their uncertainty into the cheap stage.
There's also something that happens when you draw a system. You find the questions you weren't asking. "What happens if this service is down?" "Who owns this data?" "What's the consistency model here?" These questions are trivial to answer before you build and expensive to answer after.
Before any non-trivial feature, I spend 15 minutes drawing. I'm not looking for a perfect diagram — I'm looking for surprises. Anything that makes me go "wait, how does this actually work?" is a question I want to answer before I write a single line of code.
For complex systems, I do it with the team. Not because I need help drawing boxes, but because different engineers see different problems in the same diagram. Your backend engineer sees a potential N+1 query. Your frontend engineer sees a race condition in the UI. Your SRE sees the single point of failure you designed in without realizing it.
System design interviews get a bad reputation for being disconnected from real work. In my experience, they're the most realistic interview format we have — because the actual skill being tested is the one you need most.
What's the most expensive assumption you've ever made by skipping the design phase?
by Adarsh Shankar · Senior DevOps Engineer
It was a Friday afternoon — never a good time for production incidents — when our payment service started timing out. Not because of the payments themselves, but because of what happened three steps after a payment succeeded.
When a payment went through, our service would: update the database, send a confirmation email, update an analytics dashboard, trigger a fulfillment API at a third-party warehouse, and log to an audit trail. All of this. Synchronously. In the same request thread. If the warehouse API was slow — and it was, often — the entire payment flow would wait.
This is the classic mistake. We had a synchronous system pretending the world was always fast and always available.
# The synchronous payment flow (please don't do this)
async function processPayment(payment) {
await db.save(payment) // 20ms - fine
await sendEmail(payment) // 150ms - iffy
await updateAnalytics(payment) // 80ms - okay
await notifyWarehouse(payment) // 3,000ms - WHAT
await writeAuditLog(payment) // 50ms - irrelevant now
return { success: true } // 3,300ms total. User is angry.
}The warehouse API was a third-party service we didn't control. Sometimes it responded in 200ms. Sometimes it took 5 seconds. Sometimes it timed out entirely, and our payment endpoint returned a 504 to the user even though the payment had actually succeeded.
We introduced RabbitMQ. The new payment flow: save to database, publish a payment.completed event to a queue, return success to the user immediately. Separate consumer services handle the email, analytics, warehouse notification, and audit trail asynchronously.
# The async payment flow
async function processPayment(payment) {
await db.save(payment) // 20ms
await queue.publish('payment.completed', payment) // 5ms
return { success: true } // 25ms total. User is happy.
}
# Consumers run independently, at their own pace
email-service → listens to payment.completed
analytics-service → listens to payment.completed
warehouse-service → listens to payment.completed
audit-service → listens to payment.completedThe queue is the easy part. The hard part is what happens when a consumer fails. The warehouse API times out again — what happens to that message?
Without retry logic, you either lose the message or block the entire queue. We configured:
A queue without a DLQ is a system that silently loses work under failure. DLQs aren't an edge case — they're the contract you make with every message you publish.
Payment response times dropped from an average of 1.8 seconds to under 100ms. Our payment endpoint's error rate went from ~8% (mostly warehouse timeouts bleeding through) to near zero. The warehouse service started failing independently, visibly, and without affecting users.
That Friday incident became the project that justified async architecture across our entire platform. We spent a weekend implementing it and saved every weekend after that.
If your system has a slow third-party dependency in a critical user path, what's stopping you from queueing that work?
by Adarsh Shankar · Senior DevOps Engineer
Five years ago, I thought DevOps was mostly about writing clever bash scripts and knowing which button to press when things broke. I was wrong about almost everything except the bash scripts.
I'm writing this not as a retrospective — more as the letter I wish I'd read before I started. Because nobody tells you the important parts. They teach you the tools. Nobody teaches you the culture, the burnout patterns, or the fact that the hardest part of this job isn't technical.
Early in my career, I felt vaguely guilty about automating things. Like I was cheating. If I wrote a script that did three hours of manual work in four minutes, had I actually done three hours of work?
Yes. More, actually. The script runs forever. The manual process runs once.
Automation is the purest form of respect you can show your future self and your team. Every manual step you eliminate is one less place where someone gets paged at 3AM. Every runbook you convert to a script is one less thing that requires tribal knowledge to execute correctly. Write the automation. You are not cheating. You are doing the job.
I used to treat documentation as something you wrote after everything was working, in the three minutes before the ticket was closed. Now I treat it as part of the work — not separate from it.
The most valuable documentation I've ever written is not the fancy architecture diagram. It's the runbook that says "when you see error X, do steps A, B, C, and here's why." It's the comment in the Terraform file that explains why we made a weird networking decision in 2023 that looks wrong but isn't. It's the post-incident notes that explain not just what happened, but what we learned.
If you get paged at 2AM and the runbook saves you 30 minutes of investigation, that runbook is the most valuable thing you wrote all week. Write runbooks like your future self will thank you for them. Because they will.
How a team handles on-call rotation tells you more about their engineering culture than any interview process. Ask these questions before you join:
Good on-call culture means alerts are meaningful, rotation is equitable, and the response to a bad week is to fix the system, not blame the engineer. Bad on-call culture means alert fatigue, burnout, and a team where the senior engineers have learned to ignore their phones.
I burned out at the end of year two. Not dramatically — I didn't quit, didn't have a breakdown. I just stopped caring. Started doing the minimum. Started looking at my phone every time I got an alert and hoping it would resolve itself. That's burnout — not the loud kind, the quiet kind that makes you worse at your job slowly.
What I learned: burnout in DevOps is almost always structural. Too many alerts with not enough signal. Rotation that's not actually rotated. Features prioritized over reliability without end. Being the person who "just knows" how things work and therefore gets every question.
The fix isn't a vacation. The fix is changing the structure. Better runbooks so you're not the single point of knowledge. Tighter alert thresholds. Actual rotation that includes the whole team. Management that understands reliability work has value even when nothing is broken.
DevOps is a multiplier role. You make other engineers more effective. You make systems more reliable. You make deploys boring. When you do your job well, nothing happens — and that's the point. The absence of incidents is the metric you're optimizing for.
It took me three years to stop feeling like I wasn't doing enough because nothing was on fire. Nothing on fire is the win. Learn to celebrate the quiet Fridays.
What do you wish someone had told you when you started? I'm genuinely curious — my experience isn't universal.
by Adarsh Shankar · Senior DevOps Engineer
My resume says I'm proficient in Ansible, Terraform, Helm, ArgoCD, Prometheus, Grafana, and about fourteen other tools that sound very impressive at interviews. The truth is messier and much funnier.
Let me be honest about what my actual week looks like, versus what my resume implies.
Terraform (Infrastructure as Code), Kubernetes, Helm Charts, ArgoCD for GitOps, Prometheus and Grafana for observability, Ansible for configuration management, GitHub Actions for CI/CD, AWS services from EKS to RDS to CloudFront.
This is all true. I have used all of these things. Some of them I use every day. Some of them I set up once, handed to the team, and haven't touched since. The resume doesn't make that distinction, and nobody asks.
In rough order of daily usage:
# My resume
Skills: ArgoCD, GitOps, Service Mesh, eBPF monitoring
# Reality
ArgoCD: Set it up 18 months ago, works great, haven't opened it in a month
GitOps: Fully embraced in principle, "mostly" implemented in practice
Service Mesh: Evaluated Istio, decided the overhead wasn't worth it, still on list
eBPF: Gave a talk about it. Have not used it in production.There's a difference between "I know this technology well enough to implement it" and "I use this technology every day." Resumes collapse this distinction. Interviews rarely restore it.
Here's the thing I never say in interviews: the tools that save me the most time aren't the impressive ones.
jq — I use jq to parse JSON from kubectl and AWS CLI every single day. It's never on anyone's resume. It should be. Knowing jq has saved me hours of Python scripting.
fzf — fuzzy finder for the terminal. Sounds trivial. I use it for git branches, kubectl contexts, AWS profiles. Saves minutes every day, hours every week.
A good spreadsheet — I'm serious. When I need to track something across multiple services during an incident, nothing beats a shared Google Sheet. Unglamorous. Works every time.
I'm not criticizing resumes — they're a necessary artifact and I'm not sure how else you communicate range without them. But I think our industry would benefit from more honesty about the gap between what engineers know and what they use.
The best DevOps engineers I know are brilliant at a handful of tools and comfortable with dozens of others. They know which tool to reach for and when, and they're honest about what they'd need to ramp up on.
What's the biggest gap between your resume and your actual daily work? I promise I won't tell your hiring manager.
by Adarsh Shankar · Senior DevOps Engineer
I've been using AI tools in my DevOps workflow for over a year now. Copilot for code, ChatGPT for runbooks, Claude for incident postmortems. And I have opinions — unpopular ones in some circles.
The hot take you're expecting: "AI will replace DevOps engineers." My actual take: that's not the right frame, and the people saying it haven't used these tools seriously for production work.
Boilerplate generation. Writing a Dockerfile, a basic GitHub Actions workflow, a Terraform resource block for an S3 bucket with standard settings — AI is good at this. Not because it's smarter than me, but because I've done this 200 times and I'm bored of it. Getting a 90% correct starting point that I edit into shape is faster than writing from scratch.
First-draft runbooks. If I give an AI tool the alert name and a brief description of what the service does, it gives me a reasonable draft runbook in 30 seconds. I then spend 10 minutes adding the specific commands, the actual escalation contacts, and the context that only exists in my head. This is genuinely useful.
Understanding unfamiliar error messages. "What does this PostgreSQL error mean and what are the common causes?" is exactly the kind of question AI handles well. It's fast documentation lookup with synthesis. I've stopped opening Stack Overflow for most of these.
# Things I now ask AI instead of Googling
→ "What causes SIGKILL vs SIGTERM in containers?"
→ "Explain this Kubernetes RBAC error"
→ "What's the difference between StatefulSet and Deployment?"
→ "Write a jq command to extract nested fields from this JSON"
# Things I still Google / check docs for
→ Exact API syntax for recent AWS service features
→ Kubernetes version-specific behavior changes
→ Anything where being wrong has production consequencesThis is the part nobody wants to say publicly because it sounds like you're dismissing a useful technology. I'm not. But:
AI hallucinates specific configurations. I asked an AI tool to help me configure a specific AWS ALB listener rule with a complex condition. It gave me a plausible-looking Terraform resource with a field that doesn't exist. I spent 20 minutes debugging before I checked the actual AWS docs. The AI was confidently, specifically wrong.
AI doesn't know your system. "How should I architect this?" without full context gets you generic advice that sounds good and misses the three constraints that make your situation unique. AI-generated architecture advice is a starting point, never an endpoint.
AI can't do incident response. When something is on fire, you need pattern matching against your specific system's history, intuition built from months of watching your metrics, and the ability to say "this feels like that thing that happened in March." AI has none of this. It can help you search for information. It cannot think in context.
AI is very good at the work that feels tedious. It's not good at the work that matters most — judgment under pressure, context-aware decisions, and knowing when the textbook answer is wrong for your situation.
AI tools have made me faster at maybe 30% of my work. The 30% that was repetitive, pattern-matching, "I've done this before" work. That's genuinely valuable — I can do more in a day, handle more surface area, spend less time on boilerplate.
The other 70% — the architectural decisions, the incident response, the "should we do it this way or that way and here's why our context matters" work — AI is a junior colleague who's read all the books but has no scars. I use it. I don't trust it unsupervised.
The engineers I've seen get into trouble with AI tools are the ones who forgot to stay skeptical. Who took the output and didn't verify it. Who let "good enough to compile" become "good enough for production."
Use the tools. Verify everything. Stay the engineer in the room.
What's your honest take after using AI tools in a production DevOps context? Has it changed how you work, or has it mostly been noise?