Data Engineering Expert

1.

Data Engineering & Platform Knowledge (Must)

• Strong understanding of Hadoop ecosystem: HDFS, Hive, Impala, Oozie, Sqoop, Spark (on YARN).
  
  

• Experience in data migration strategies (lift & shift, incremental, re-engineering pipelines).
  
  

• Knowledge of Databricks architecture (Workspaces, Unity Catalog, Clusters, Delta Lake, Workflows).
  
  

2.

Testing & Validation (Preferred)

• Data reconciliation (source vs.
  
  target).
  
  

• Performance benchmarking.
  
  

• Automated test frameworks for ETL pipelines.
  
  

3.

Databricks-Specific Expertise (Preferred)

• Delta Lake: ACID transactions, time travel, schema evolution, Z-ordering.
  
  

• Unity Catalog: Catalog/schema/table design, access control, lineage, tags.
  
  

• Workflows/Jobs: Orchestration, job clusters vs.
  
  all-purpose clusters.
  
  

• SQL Endpoints / Databricks SQL: Designing downstream consumption models.
  
  

• Performance Tuning: Partitioning, caching, adaptive query execution (AQE), photon runtime.
  
  

4.

Migration & Data Movement (Preferred)

• Data migration from HDFS/Cloudera to cloud storage (ADLS/S3/GCS).
  
  

• Incremental ingestion techniques (Change Data Capture, Delta ingestion frameworks).
  
  

• Mapping Hive Metastore to Unity Catalog (metastore migration).
  
  

• Refactoring HiveQL/Impala SQL to Databricks SQL (syntax differences).
  
  

5.

Security & Governance (Nice to have)

• Mapping Cloudera Ranger/SSO policies Unity Catalog RBAC.
  
  

• Azure AD / AWS IAM integration with Databricks.
  
  

• Data encryption, masking, anonymization strategies.
  
  

• Service Principal setup & governance.
  
  

6.

DevOps & Automation (Nice to have)

• Infrastructure as Code (Terraform for Databricks, Cloud storage, Networking).
  
  

• CI/CD for Databricks (GitHub Actions, Azure DevOps, Databricks Asset Bundles).
  
  

• Cluster policies & job automation.
  
  

• Monitoring & logging (Databricks system tables, cloud-native monitoring).
  
  

7.

Cloud & Infra Skills (Nice to have)

• Strong knowledge of the target cloud (AWS/Azure/GCP):

o Storage (S3/ADLS/GCS).

o Networking (VNETs, Private Links, Security Groups).

o IAM & Key Management.

9.

Soft Skills

• Ability to work with business stakeholders for data domain remapping.
  
  

• Strong documentation and governance mindset.
  
  

• Cross-team collaboration (infra, security, data, business).