Responsibilities

• Lead the development and optimization of data pipelines and ETL processes using Databricks.
• Collaborate with teams who work on Integrating diverse data sources and apply business rules for predictive modeling.
• Collaborate with internal teams to generate the Next Best Actions (NBA) for field representatives.
• Have a deep understanding and knowledge of AI/ML and ML Ops platforms to enhance data processing capabilities.
• Work with commercial teams to process and analyze structured data inputs.
• Ensure seamless handover and collaboration with existing teams.
• Mentor junior engineers and provide technical leadership.

Must-Have Skills

• Python – 5 years
• SQL – 5 years
• Machine Learning
• Apache Spark – 5 years
• Data Engineer – 5 years

Required Skills and Experience

• 5-10 years of experience in Data Engineering and related streams.
• Proficient in SQL, Python, and Spark.
• Extensive experience with Databricks on AWS.
• Strong understanding of AI/ML concepts and ML Ops platforms, particularly the Arise platform.
• Proven ability to develop and optimize data engineering pipelines.
• Experience with predictive modeling and business rule application.
• Excellent collaboration and communication skills.
• Demonstrated ability to lead projects and mentor junior engineers.

Preferred Qualifications

• Familiarity with commercial data processing and forecasting models.
• Background in handling large-scale data operations and integration.