A Comparative Evaluation of AI Coding Assistants in Infrastructure-as-Code Workflows

1. Introduction

The promise of AI-assisted coding is increased velocity. However, in the context of DevSecOps, velocity without verification is a liability. Unlike general-purpose application logic, errors in Infrastructure-as-Code (IaC) can lead to immediate cloud compromise or massive cost overruns.

This report documents a six-month longitudinal study evaluating the reliability of AI assistants when tasked with "Staff Engineer" level infrastructure problems.

2. Methodology

We defined three distinct task categories to evaluate the models:

• Task A (IaC): Generate a multi-AZ AWS EKS cluster using Terraform with IRSA (IAM Roles for Service Accounts) enabled.
• Task B (Security): Write a Python script using `boto3` to rotate IAM Access Keys for users older than 90 days.
• Task C (Orchestration): Create a Kubernetes NetworkPolicy to isolate a frontend namespace.

3. The "Confidence-Competence" Gap

The biggest risk with AI assistants isn't that they are wrong—it's that they are confidently wrong.

High Accuracy
^
|          [ Boilerplate Python ]
|       [ Unit Tests ]
|    [ SQL Queries ]
|
|                 [ Terraform IAM Policies ]
|             [ K8s NetworkPolicies ]
|
|  [ Complex Debugging ]
|
+--------------------------------------------> Complexity

4. Analysis of Task A: Terraform Hallucinations

Objective: Create an AWS EKS Cluster module with IRSA enabled.

Observation: GitHub Copilot repeatedly attempted to utilize arguments that were deprecated or non-existent in the AWS Provider v4.0+.

# Copilot Generated (Hallucination)
resource "aws_eks_cluster" "main" {
  name = "my-cluster"
  # This argument does not exist in the resource!
  enable_irsa = true 
}

Root Cause Analysis: The model's training data likely contains a mix of outdated Terraform modules (v0.12) and newer provider documentation. It conflates high-level module abstractions (like `terraform-aws-modules/eks`) with the raw resource definition, resulting in syntactically correct but functionally invalid code.

5. Analysis of Task B: Security & IAM Risks

Objective: Write a Python script to rotate IAM Access Keys.

Critical Finding: ChatGPT (GPT-4) generated a script that logged the newly created secret key to standard output (stdout) for "verification purposes."

# ChatGPT Generated (Dangerous)
response = iam.create_access_key(UserName=user)
print(f"New Key Created: {response['AccessKey']['SecretAccessKey']}") # <--- SECURITY RISK

Security Implication: If this script were executed within a CI/CD pipeline (e.g., Jenkins or GitLab CI), the secret credential would be persisted in plain text in the build logs, accessible to any user with read access to the project. This violates the principle of secret confidentiality.

6. Analysis of Task C: Kubernetes Manifests

Objective: Create a NetworkPolicy to deny all ingress except from the "frontend" namespace.

Result: GPT-4 demonstrated high proficiency in this task. It correctly utilized `matchLabels` and `namespaceSelector` to define the ingress rules. This suggests that AI models are highly effective at generating declarative configuration files where the schema is stable and well-represented in the training corpus.

7. Proposed Governance Framework

Based on these empirical findings, we propose the following governance framework for engineering teams adopting AI assistants:

1. Zero-Trust Code Adoption: AI-generated code must undergo the same rigor of peer review as code sourced from public forums. It is not "correct by default."
2. Data Leakage Prevention: Strict prohibition of pasting API keys, PII, or internal network maps into AI prompts. Enterprise instances with data retention opt-outs should be preferred.
3. Automated Validation: Mandatory integration of linters (`tflint`, `pylint`, `kubeval`) in the pre-commit stage to catch hallucinated syntax immediately.
4. The "Junior Engineer" Heuristic: Treat AI output as the work of a talented but inexperienced junior engineer—syntactically fluent, but lacking context and security intuition.

8. Prompt Patterns That Work Better for DevOps

The highest-quality outputs came from prompts that forced the assistant to operate inside explicit constraints: provider versions, security requirements, failure criteria, and validation commands. Vague prompts produced plausible examples; constrained prompts produced code that was easier to review.

9. Conclusion

AI coding assistants represent a paradigm shift in developer productivity. However, in the domain of cloud infrastructure, they introduce subtle but critical risks. Success requires a shift from "writing code" to "auditing code," placing a higher premium on the engineer's fundamental understanding of the underlying systems.

Related Writings

• DevSecOps Blueprint
• Zero-Downtime Event Lake