Design, develop, and maintain real-time data streaming pipelines using Spark Streaming or Azure Functions.
- Load, merge, and process machine logs from Kafka, ensuring efficient data flow and transformation.
- Integrate processed data into Redis cache and send it to the data lake for long-term storage and analysis.
- Implement and optimize data processing solutions using Python.
- Apply software engineering best practices, including code reviews, version control, and continuous integration/deployment (CI/CD).
- Collaborate with cross-functional teams to understand requirements and deliver high-quality data solutions.
Proven experience in real-time data streaming using Spark Streaming or Azure Functions. Strong proficiency in Python for data processing and automation tasks. Hands-on experience with Kafka for data ingestion and message queuing. Familiarity with Redis for caching and fast data retrieval. Knowledge of data lake architectures and best practices for data storage and retrieval. Solid understanding of software engineering principles, including design patterns, testing, and documentation. Strong problem-solving skills and ability to work in a fast-paced, collaborative environment.
Preferred Qualifications:
Experience with cloud platforms such as Azur Knowledge of other programming languages and frameworks related to data engineering. Familiarity with data modeling, ETL processes, and data lake concepts.