Data Engineer MS Fabric PySpark

SQLI

Job description

As a Data Engineer, you will join our data team to participate in strategic data analysis projects and the implementation of large-scale data processing solutions. You will be responsible for designing, optimizing and developing data pipelines on cloud platforms, with a focus on the use of PySpark and Azure Data Factory.

Your main missions will be:

Design, develop, and maintain large-scale data pipelines using PySpark.

Enable data flows with Azure Data Factory, integrating various data sources and destinations (SQL, Data Lakes, APIs, etc.).

Ensure the integration of data from various systems in a cloud environment (Azure).

Optimize the performance of data processing (SQL, PySpark, etc.).

Collaborate with business teams, data analysts, and data scientists to understand needs and deliver effective and scalable solutions.

Participate in the definition of the data architecture and governance strategy.

Set up and monitor data processing jobs (error management, performance management).

Document the solutions implemented and follow the best development practices.

Participate in the continuous improvement of existing processes and tools.

Qualifications

Education and experience:

Bac +5 in computer science, data science, or an equivalent field.

Minimum of 3 years of experience in a similar position, with proven experience in development on PySpark and Azure Data Factory.

Technical skills:

Proficiency in PySpark for massive data processing in a distributed environment.

Significant experience in setting up and managing Azure Data Factory for building and orchestrating data pipelines.

Strong experience in relational and non-relational databases (SQL, NoSQL).

Knowledge of data management tools in the Azure ecosystem (Azure Data Lake, Azure SQL Database, Azure Blob Storage, etc.).

Working knowledge of Azure cloud services and their use in data processing solutions (Azure Databricks, Azure Synapse Analytics, etc.).

Experience with version control tools (Git, Azure DevOps, etc.) and software development best practices.

Knowledge of big data concepts and performance optimization in distributed environments.

Personal Skills:

Autonomy, rigor and attention to detail.

Excellent ability to solve complex technical problems.

Good communication skills and ability to work in a team.

Analytical skills and the ability to understand business needs to propose appropriate solutions.