Senior Databricks Engineer - AI & Data Platform

• Career Progression
• Autonomy

Sobre o nosso cliente

An organisation in the Technology & Telecoms industry located in Lisbon Center

Descrição

Key Responsibilities

• AI Application Enablement: Support the development and deployment of AI-powered applications, including LLM/GenAI workflows, RAG pipelines, and integration with Mosaic AI tooling.
• Data Platform Foundation: Design, build, and maintain the core Databricks Lakehouse platform, including workspace configuration, Unity Catalog setup, cluster policies, security, and governance.
• Data Engineering: Develop and optimize production-grade data pipelines, delivering clean, reliable datasets for analytics and AI workloads.
• Solution Architecture: Evaluate requirements, produce reference architectures, and recommend patterns for data ingestion, transformation, storage, and consumption that align with AI use cases.
• Standards & Best Practices: Establish engineering standards for code quality, CI/CD (Databricks Asset Bundles, Repos), testing, monitoring, and cost optimization.
• Collaboration: Work cross-functionally with data analysts, cloud/infra teams, and business units to translate requirements into scalable technical solutions.

Perfil ideal

• 5+ years in data engineering or platform engineering roles.
• 3+ years hands-on experience with Azure Databricks (workspace administration, Delta Lake, Spark, Unity Catalog).
• Strong proficiency in Python and SQL.
• Databricks workspace security: Proven experience implementing and managing workspace-level security, including network security (Private Link, VNet injection, IP access lists), identity and access management, cluster policies, secrets management, and RBAC at workspace, catalog, and object levels.
• Experience building and operating production data pipelines at scale.
• Experience with Mosaic AI (AI Playground, AI Functions, Agent Framework)
• Familiarity with GenAI/LLM application patterns (RAG, prompt engineering, vector databases) is highly desirable.

Nice to Have



Databricks certifications (Data Engineer Associate/Professional, Gen AI Engineer Associate).

· Experience with Unity Catalog governance at enterprise scale (metastore design, data access policies, lineage, audit logging).

· Hands-on experience with SAP integration , extracting data from S/4HANA into a Lakehouse platform.

· Experience leading or participating in data migration from SAP BW/4HANA to Databricks, including InfoProvider mapping, transformation logic re-implementation, data validation, and cutover planning.

· Exposure to AI/ML workflows: model training infrastructure, feature stores, MLflow, or equivalent MLOps tooling - and ability to prepare SAP-sourced datasets for downstream ML consumption.

• Strong communication skills, ability to produce architecture diagrams, design documents, and present trade-offs to technical and non-technical audiences.

Vantagens

• Annual salary: €65000 - €80,000 (paid in 14 monthly instalments).
• annual bonus.
• Meal allowance of €7 per day via meal card.
• Single cover private health insurance.
• Hybrid working model with a minimum of 2-3 days in the office.
• Permanent contract.