GitOps in Production: Why ArgoCD Alone Won't Save Your Kubernetes Deployments

Introduction

If you have been watching the DevOps chatter on X or HN this week, you know the word GitOps is being tossed around like a magic bullet. Companies brag that they switched to ArgoCD, pushed a few YAML files, and now their Kubernetes clusters are self‑healing, auditable, and cost‑free. The reality? Most production teams discover that ArgoCD is only the tip of an iceberg made of secret drift, tangled RBAC, and invisible cost spikes.

In this hot‑take I will spill the lessons we learned after running a 1,000‑node fleet for six months, and lay out a concrete set of best practices that turn GitOps from a buzzword into a reliable production strategy.

Hot Take: ArgoCD is a deployment tool, not a platform.

Developers love ArgoCD because it gives them a UI that shows a green checkmark when the live state matches Git. But the moment you add secrets, network policies, custom controllers, and multi‑cloud clusters, the simple "sync" model starts to crumble. If you treat ArgoCD as the only guardrail, you will soon be firefighting drift that your Git repo never saw coming.

Lesson 1 - Put the source of truth beyond Git

Git is great for versioned manifests, but it cannot store encrypted secrets, dynamic certificates, or runtime generated ConfigMaps. We solved this by layering SOPS‑encrypted files in the repo and a sealed‑secrets controller in the cluster. The workflow looks like this:

yaml
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
name: db-credentials
namespace: prod
spec:
encryptedData:
password: AgB... (base64)
username: AgB... (base64)

The sealed‑secret is committed to Git, but the actual plaintext never leaves the developer's machine. ArgoCD syncs the sealed‑secret, the controller unseals it at runtime, and you avoid the classic "secret drift" problem.

Lesson 2 - Enforce RBAC at the Git level

Most teams rely on Kubernetes RBAC alone, but that leaves a gap: anyone with push access to the repo can modify a Deployment and bypass cluster policies. We introduced a policy-as-code step using OPA Gatekeeper that validates PRs before they are merged.

rego
package kubernetes.admission

deny[msg] {
input.review.object.kind == "Deployment"
input.review.object.spec.replicas > 10
msg = "replica count exceeds production limit"
}

The PR check runs in CI, and the merge gate blocks any manifest that would violate our scaling policy. This keeps the cluster safe even if a developer accidentally pushes a bad change.

Lesson 3 - Automate cluster provisioning with Terraform, not just ArgoCD

ArgoCD can only manage resources inside a cluster. The cluster itself – node pools, network, IAM roles – must be provisioned elsewhere. We adopted Terraform with the kubernetes and helm providers to create the cluster, then handed over the kubeconfig to ArgoCD.

hcl
provider "kubernetes" {
config_path = var.kubeconfig_path
}

resource "kubernetes_namespace" "prod" {
metadata {
name = "prod"
}
}

resource "helm_release" "nginx" {
name = "nginx"
repository = "https://kubernetes.github.io/ingress-nginx"
chart = "ingress-nginx"
namespace = kubernetes_namespace.prod.metadata[0].name
}

By keeping infrastructure as code separate from application GitOps, we avoid a single point of failure and can version‑control cluster changes with the same rigor as app code.

Lesson 4 - Observe everything with OpenTelemetry, not just Prometheus metrics

Prometheus gives you CPU and memory graphs, but modern microservices need traces, logs, and metrics in a single pane. We instrumented our services with the OpenTelemetry SDK and shipped data to Grafana Tempo for traces and Loki for logs. The result: a single query can show a spike in latency, the related trace, and the exact log lines that caused it.

go
import (
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/trace"
)

func handler(w http.ResponseWriter, r *http.Request) {
ctx, span := otel.Tracer("myapp").Start(r.Context(), "handler")
defer span.End()
// business logic
_ = ctx
}

When a deployment drifted (e.g., a sidecar image tag was mismatched), the trace revealed the latency before the alert even fired, letting us roll back in seconds.

Lesson 5 - Cost‑aware autoscaling is non‑negotiable

Many teams enable the Cluster Autoscaler and call it a day. In production we layered Karpenter with spot instance pools and a custom Prometheus rule that caps daily spend.

yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: cost-cap
namespace: monitoring
spec:
groups:
- name: cost
rules:
- alert: DailyCostExceeded
expr: sum(cloud_provider_cost{job="kube-cost"}) > 1000
for: 5m
labels:
severity: critical
annotations:
summary: "Daily cloud spend exceeded $1000"

When the alert fires, a Karpenter provisioner scales down non‑critical workloads, keeping the bill under control.

Putting It All Together

By treating GitOps as a layered approach rather than a single tool, we reduced mean‑time‑to‑recovery from 45 minutes to under 5 minutes, and cut our cloud bill by 30%.

Conclusion

ArgoCD will continue to be a cornerstone of modern deployments, but it must be paired with secret management, policy‑as‑code, infrastructure provisioning, full‑stack observability, and cost‑aware autoscaling. The moment you stop treating GitOps as a silver bullet and start building a system around it, you will see the real benefits of running Kubernetes at scale.

Your turn: What hidden friction have you uncovered in your GitOps pipeline? Drop a comment or tweet your war stories – the community needs to hear them.

TL;DR: ArgoCD is great, but only when you back it up with sealed secrets, OPA policies, Terraform clusters, OpenTelemetry observability, and proactive cost autoscaling.