Reach peak performance | IT consulting and software engineering backed by our expertise in Dev Experience, ML and Scala
Staff Data Engineer
Location
New York
Posted
4 days ago
Salary
Not specified
Job Description
Job Requirements
- Proven experience building web crawling or large-scale data systems from scratch
- Strong architectural skills in designing scalable, fault-tolerant distributed systems
- Track record leading complex technical initiatives and driving architecture direction for teams
- Demonstrated ability to evolve production systems incrementally while maintaining reliability
- Experience mentoring engineers at all levels and promoting a collaborative culture
- Deep background in large-scale data engineering (terabytes daily)
- Hands-on experience with cloud data warehouses (BigQuery, Snowflake)
- Experience with Apache Kafka, Kubernetes (GKE/EKS), and orchestration tools (Airflow)
- Familiarity with multi-cloud environments (GCP + AWS)
- Expertise in designing and operating ETL/ELT pipelines
- Deep expertise in web crawling technologies and advanced scraping (Scrapy or similar)
- Experience in extracting structured/unstructured web data and SERP extraction
- Knowledge of proxy infrastructure management, anti-bot detection, and ethical crawling
- Familiarity with crawling vendors and AI/LLM-based extraction approaches
- Support the VirtusLab U.S. and international teams by lending senior technical expertise to client-facing activities, including technical discovery sessions, workshops, and solution architecture
- Conduct requirements analysis and solution discovery, identifying business and technical needs
- Provide technical consulting and advisory services, recommending appropriate data architectures aligned with customer goals
- Prepare and review technical sections of commercial offers, including solution descriptions, statements of work (SoWs), project estimates, timelines, and delivery models
Benefits
- self-development opportunities
- good working conditions
Related Guides
Related Categories
Related Job Pages
More Data Engineer Jobs
Relativity Archiving Analyst
Contact Government ServicesContact Review prides itself on finding high-quality, high-accountability, barred attorneys specifically tailored to the needs of our project. Assists with document review, privilege review, expert testimony, legal research, and foreign language translation Fosters a culture where every team member sees themselves as an extension of the project's team Looks for ways to improve efficiency and streamline workflows
CGS is seeking a Relativity Archiving Analyst, who will be responsible for vetting Relativity workspaces and file share folders and archiving or purging them. File shares will be moved to archive locations. Relativity workspaces will be archived using both Relativity ARM and a fl...
Lead Microsoft Data Engineer
Procentrix, LLCDelivering practical solutions to solve complex challenges with an eye on maximizing customers' current IT investments
The Lead Azure Data Engineer is responsible for migration, governance, and data integration for the implementation of new Microsoft Cloud based document routing and task management system. They will ensure alignment with the technical, security, and document management requiremen...
Data Engineer I
Centene CorporationTransforming the health of the communities we serve, one person at a time.
The role involves developing and operationalizing data pipelines for data ingestion, transformation, validation, and optimization to support reporting and advanced analytics consumption. This includes designing and implementing standardized data management procedures and developing high-performance data structures for business intelligence.
Contract Data Engineer
SilvurThe only retirement planning platform exclusively built for those over 50. Get your Retirement Score.
Contract Data Engineer optimizing SQL for retirement decision-making tools
