Data Engineer

Core42

Overview

About Us

Core42, a leader in AI-powered cloud and digital infrastructure, is driving transformative technology solutions globally. Leveraging advanced resources and partnerships, Core42 empowers clients to harness sovereign AI infrastructure, especially in sectors with stringent regulatory needs. With a mission to redefine digital transformation, we combine sovereign capabilities with scalable, high-performance compute infrastructure, positioning itself at the forefront of AI innovation in the Middle East and beyond.

The opportunity

We are seeking a highly skilled and versatile Data Engineer with expertise in building robust data solutions using the Microsoft Data Platform and hands-on experience in AI/ML model development. This role focuses on designing, developing, and maintaining scalable and efficient data pipelines and architectures while also contributing to the development and deployment of AI scoring engines and predictive models to support advanced analytics.The ideal candidate will have deep expertise in Azure cloud services, data governance, data modeling, and advanced tools like Microsoft Fabric and Databricks, with a complementary focus on AI/ML workflows and MLOps. This position offers the opportunity to work at the intersection of data engineering and artificial intelligence to drive meaningful business outcomes

Responsibilities

Your key responsibilities

Primary Responsibilities Microsoft Data Platform Expertise

• Design & Develop Data Pipelines:
• Build, optimize, and maintain ETL/ELT pipelines to ingest, transform, and process large volumes of structured and unstructured data from diverse sources.
• Leverage Azure Data Factory, Azure Data Lake, Azure Synapse
• Analytics, Microsoft Fabric, or Databricks for scalable data integration and transformation.
• Data Modeling & Architecture:
• Design and implement data models and schemas optimized for analytics, machine learning, and AI-driven decision-making.
• Create and manage data warehouses, data lakes, and lakehouses to support data analytics and AI workloads.
• Data Governance & Security:
• Ensure data quality, compliance, and security by implementing governance frameworks and leveraging tools like Microsoft Purview.
• Enforce data security protocols, including role-based access control, data masking, and encryption.
• AI/ML Data Integration:
• Collaborate with data scientists to integrate data pipelines with machine learning workflows, enabling seamless training and deployment of models.
• Performance Optimization:
• Monitor and optimize the performance of data pipelines and cloud resources to ensure high availability, scalability, and cost efficiency.
  
  

Secondary Responsibilities AI/ML & Scoring Engine Development

• Build AI Scoring Engines:
• Develop and implement AI scoring engines to automate decision-making processes, such as fraud detection, customer segmentation, and recommendation systems.
• Data Preparation for AI/ML Models:
• Partner with data scientists to prepare and preprocess data for machine learning models, including handling missing values, scaling, and feature engineering.
• AI/ML Model Development:
• Contribute to the development of predictive and prescriptive models using frameworks like scikit-learn, TensorFlow, PyTorch, and Azure ML Studio.
• Model Deployment and Integration:
• Deploy machine learning models and scoring engines to production environments using Azure Machine Learning, integrating real-time and batch workflows.
• MLOps Implementation:
• Build and maintain MLOps pipelines for versioning, monitoring, and retraining AI models in production, ensuring continuous improvement.
• AI-Driven Insights:
• Support the integration of AI models with business applications to deliver actionable insights and improve operational efficiency.
  
  

Qualifications

• Education: Bachelor s or Master s in Computer Science, Data Engineering, Data Science, or related field.
• 6 10 years of experience in data engineering, with a strong focus on Microsoft Data Platform technologies.
• 4+ years of hands-on experience in AI/ML model development, including building and deploying scoring engines.
  
  

Technical Skills

Microsoft Data Platform

• Advanced knowledge of Azure Data Factory, Azure Synapse Analytics, Azure
• Data Lake, Microsoft Fabric, Databricks, and SQL Server.
• Experience with Azure DevOps for CI/CD pipelines in data engineering workflows.
  
  

AI & Machine Learning

• Proficient in Python and SQL for data engineering and ML tasks.
• Hands-on experience with AI/ML libraries such as scikit learn, TensorFlow, PyTorch, or Azure ML Studio.
  
  

Big Data Processing

• Experience with distributed data processing frameworks like Apache Spark forhandling large-scale datasets.
  
  

Data Governance Tools

• Experience with tools like Microsoft Purview for data cataloging, governance, andcompliance.
  
  

AI/ML Deployment & MLOps Practices

• Strong understanding of MLOps best practices, including model versioning, monitoring, and retraining.
  
  

Certifications

• Azure Data Engineer Associate
• Azure AI Engineer Associate
• Databricks Certified Data Engineer Associate
• Microsoft Fabric Certification
• Experience with data lakehouse architectures and advanced tools like Microsoft Fabric.
• Exposure to advanced AI domains such as NLP, computer vision, or time series forecasting.
• Familiarity with containerization tools like Docker and orchestration tools like Kubernetes.