• Design, implement, and maintain web scraping pipelines for a wide variety of websites and data sources. • Build scrapers using tools and frameworks such as Selenium, Playwright, BeautifulSoup, Scrapy (and similar libraries) with a focus on reliability, performance, and maintainability. • Create automated workflows for scraping and data processing: • Containerize scraping jobs (e.g., using Docker). • Deploy and orchestrate them in the cloud (e.g., AWS, GCP, Azure). • Configure scheduling (e.g., run daily/weekly/hourly) and dependency management. • Implement monitoring, alerting, and logging: • Capture detailed logs for each job run. • Track job statuses and failures. • Implement notifications/alerts when a scraper breaks or a website changes. • Handle anti-bot measures (proxies, captchas, rate limits) and design scrapers that are resilient to layout and structure changes. • Work closely with data engineering / product / ML teams to understand data requirements and ensure data quality. • Utilize LLMs (Large Language Models) to: • Parse and extract structured information from messy HTML or semi-structured content. • Increase robustness of scrapers to frequent UI/DOM changes. • Prototype new scraping / extraction strategies using LLM APIs. • Write clean, well-tested, and well-documented code, and contribute to best practices, code reviews, and tooling for the team. • Continuously improve the scraping platform, including performance optimizations, standardization, and reusability of components.