Software Engineer, Data Platforms

Software Engineer, Data Platforms
Middesk, Inc — New York, NY

Middesk, Inc., New York, NY seeks to hire Software Engineers, Data Platforms (multiple positions available) to perform the following duties: 1\. Building and maintaining batch and streaming data pipelines critical to Middesks data infrastructure using tools such as Airflow, BigQuery, Dataflow, Dataproc, and VertexAI, and building and managing ETL data pipelines to power machine learning models and APIs. 2. Ensuring the reliability, scalability, and maintainability of systems across the stack.3\. Building, scaling, and extending web scraping platforms and capabilities for real-time data acquisition, and building and managing data gathering web-scrapers in a Python-based environment.4\. Updating and maintaining the interface layer between our application and data platform stack to improve performance as we continue to scale our platform and datasets, including developing RESTful APIs using FastAPI to support data validation, ingestion, and entity resolution for multiple enterprise customers.5\. Collaborating cross-functionally with multiple teams to gather product requirements and conducting design reviews to ensure alignment.6\. Contributing to the continuous improvement of development processes and methodologies, including leveraging open-source tools (PySpark, Delta) to improve data quality, pipeline health and monitoring.7\. Researching and adopting new technologies and open source implementations. May telecommute Requires a bachelors (or foreign educ. equiv.) Degree in Computer Science, Computer Engineering, or a closely related field plus three (3) years of experience in the job offered or related occupation. Experience must include: a. Building and managing ETL data pipelines to power machine learning models and APIsb. Building and managing data gathering web-scrapers in a Python-based environmentc. Developing RESTful APIs using FastAPI to support data validation, ingestion, and entity resolution for multiple enterprise customersd. Leveraging open-source tools (PySpark, Delta) to improve data quality, pipeline health and monitoring To apply, send resume to hr@middesk.com & refer to SE