Senior Data Engineer

3 сентября 2018    38
Откликнуться

Intro

Daltix is a startup from Ghent (BE) with offices in Boom (BE) and Lisbon (PT) bringing real-time insights to the world of retail. We have developed a set of tools to gather, process and analyze e-commerce data from webshops. Recently we've got an office upgrade and are located in a beautiful office near Cais Do Sodre. In the middle of the city center where "work hard & play hard" just all makes more sense.

Responsibilities & skills

We are looking for a senior profile to extend and maintain the data pipeline that is at the heart of our business. We are a data-driven company which collects and processes more than 500GB of raw data daily. We leverage big data technologies such as Spark on AWS EMR to crunch these volumes of data and make it queryable.

In this role you will ensure uptime, quality, and build upon our mission-critical Spark and Scala/Python framework that turns the HTML of millions of retail web pages into structured JSON, CSV & ORC formats.

In order to do this, you’ll need to jump head first into this state of the art data lake designed and implemented by Belgium’s top data engineering firm DataMinded.

As our data engineering tech lead you will be responsible for the following topics:

  • DataLake expansion & maintenance.
    • Required skills: Spark, Scala, AWS (EMR, Lambda, Elastic Search, Athena, …) , Apache Airflow.
  • Infrastructure setup for data processing pipelines.
    • Required skills: EMR, Spark, Airflow, Automation, Docker, …
  • Design and build big data processing architectures to build further use-cases (such as AI and machine learning) upon our data.
    • Required skills: AWS solutions/architecture design, Python, cost-awareness.
  • Data Mart/Warehouse design to make our data more accessible for BI tools, marketing and analytics.
    • Required skills: SQL database/data warehouse design, Python, data modeling.

On top of all this you’ll make sure that Daltix stays competitive in terms of data processing by using the latest & most suitable data processing technologies throughout our stack.

Main requirements

  • You have a master in computer engineering (or relevant computer engineering degree).
  • 5 years of relevant experience of which 3 years of programming experience in Python.
  • Deep understanding of databases/data warehouses/data lakes.
  • Experience with big data technologies (such as Hadoop, Spark, Airflow, Cassandra, Elasticsearch).
  • Able to lead data engineering projects as a tech lead.
  • Highly proficient in spoken and written English.
  • You are passionate about data engineering and you can demonstrate a hands on experience with data analysis tools.

Nice to have

  • Ideally you also…

    • Have experience building on top of Amazon Web Services.
    • Have some programming experience with Scala.
    • Have a deep understanding of cloud possibilities and limitations in the areas of distributed systems, load balancing and networking, massive data storage, and security.
    • Get energy from working in a highly complex and challenging startup environment with a high tech product.

    *

    Daltix’ offers a competitive wage and a young, dynamic and international atmosphere to work in.

    You will also receive the possibility to work from home if you prefer (even if you live in Lisbon).


    Besides developing your technical skills you will also have the opportunity to grow into the following skillsets:

    • Technical/architectural lead.
    • SW project management.
    • Team leading & coaching.

Perks

  • An open-minded working environment
  • Team dinners
  • Belgian beers
  • Good vibes only
  • Bad jokes

Подписывайтесь на наш телеграм-канал @remotelist, чтобы всегда быть в курсе новых вакансий! Дайджесты с новыми вакансиями появляются каждые 2-3 часа.

Еженедельная рассылка топ-15 самых просматриваемых вакансий сайта. Письмо приходит каждое воскресенье.