Data Engineer

Job description

tl;dr: Data Engineer; post-acquisition profitable health/adtech; processing 20TB+ daily at near-realtime speed; Python, Spark; remote, up to 150k USD base, we can talk higher figures and EU/UK/US employment PulsePoint sits at the intersection of healthcare and adtech. We help brands and agencies interpret the hard-to-read signals across the health journey and unify these digital determinants of health with real-world data to produce the most dimensional view of the customer. We are 300+ and growing, post-acquisition business and one of the leading players in the US healthcare ad market. We're looking for a Data Engineer to join a team, playing a key role at a technology company experiencing exponential growth. The data pipeline processes over 80 billion impressions a day (20TB+ of data, 220TB uncompressed), powering reports, budget updates, and optimization engines against extremely tight SLAs, with stats and reports delivered as close to real-time as possible. Team responsibilities - Install, maintain, and monitor Kafka, Hadoop, Presto, and RDBMS systems. - Ingest, validate, and process internal and third-party data. - Create, maintain, and monitor data flows in Hive, SQL, and Presto for consistency, accuracy, and lag time. - Maintain and enhance the framework for jobs (primarily aggregate jobs in Hive). - Build Kafka consumers using Spark Streaming for…