Key Responsibilities
- Provision and configure AWS infrastructure components required for Databricks (e.g., VPCs, subnets, IAM roles, security groups, S3 buckets, KMS, etc.).
- Set up and configure Databricks Workspaces and enable Unity Catalog in Non-Prod environments following organizational standards.
- Ensure proper configuration of access policies, identity federation, network security, and data governance controls.
Infrastructure as Code (Terraform + DAB):
- Design, develop, and maintain Terraform scripts to automate provisioning of AWS infrastructure for Databricks and Unity Catalog using DAB (Data Access and Billing).
- Create reusable Terraform modules for consistent deployment across environments.
- Collaborate with architecture and security teams to ensure compliance with best practices and cloud governance policies.
- Automate resource creation, configuration, and teardown using CI/CD pipelines (e.g., GitHub Actions, Jenkins, or AWS CodePipeline).
General Responsibilities
- Document manual setup steps and transition plan to Terraform-based deployments.
- Troubleshoot issues related to resource provisioning, workspace configuration, and catalog access.
- Support multiple Non-Prod environments, ensuring consistency and stability.
- Coordinate with data engineering and platform teams for integration and deployment readiness.
Required Skills & Qualifications
- Good Handson experience in AWS infrastructure provisioning, including VPC, IAM, EC2, S3, and KMS.
- Strong hands-on experience with Databricks on AWS, including Workspace setup and Unity Catalog enablement.
- Proficiency in Terraform and infrastructure automation for cloud services.
- Familiarity with DAB (Data Access and Billing) concepts and configuration for Databricks.
- Understanding of cloud networking, IAM policies, cross-account access, and security best practices.
- Experience in both manual infrastructure setup and automated IaC implementation.
- Solid knowledge of Git-based workflows and version control.