Lead Python Engineer, Data Infrastructure

Backend EngineerSoftware EngineerFull TimeRemoteTeam 51-200H1B No SponsorCompany SiteLinkedIn

Location

United States

Posted

4 days ago

Salary

Not specified

Bachelor Degree5 yrs expEnglishJava ScriptPostgre SQLPython

Job Description

• Lead the design and implementation of robust, efficient, and large-scale web scraping platforms using Python and associated frameworks. • Mentor junior developers, and provide technical guidance. Conduct code reviews to ensure the delivery of high-quality, maintainable code. • Develop sophisticated strategies to handle and bypass advanced anti-bot countermeasures like CAPTCHAs, Cloudflare, and IP blocking, while ensuring all practices adhere to legal and ethical guidelines and website terms of service. • Collaborate with data analysts and data engineers to define data requirements and ensure seamless integration of scraped data into databases. • Optimize scrapers for speed, performance, and stability; set up real-time monitoring and alerting systems to quickly detect and resolve failures or site changes. • Create clear technical documentation and communicate effectively with cross-functional teams and stakeholders to ensure alignment and manage expectations.

Job Requirements

  • 5+ years of experience in Python development, with prior experience in a leadership or senior role.
  • Strong programming skills and deep knowledge of Python data structures and libraries.
  • Solid understanding of HTML, CSS, JavaScript, HTTP protocols, cookies, headers, and DOM manipulation.
  • Experience with data cleaning, processing, and storage in various database systems like PostgreSQL.
  • Strong problem-solving and analytical skills.
  • Excellent attention to detail and data accuracy.
  • Effective communication skills for collaborating with cross-functional teams.

Benefits

  • A small, collaborative, and fast-moving team where your contributions will have an outsized impact.
  • The chance to work on meaningful problems in regulatory technology.
  • Remote-first culture with flexibility and autonomy.
  • Recognition in the regtech space for our innovation and customer value.

Related Job Pages

More Backend Engineer Jobs

Full TimeRemoteTeam 51-200

The Staff Engineer will serve as the technical authority for core components, driving architecture, design, and end-to-end delivery of complex initiatives while ensuring documentation and continuous improvement of their domain. Responsibilities include leading major feature designs, owning component quality across the stack, mentoring junior staff, and participating in strategic architectural discussions.

PythonGoJavaC++System DesignDistributed Systems
United States
$150K - $180K / year
Full TimeRemoteTeam 51-200

The Senior Engineer develops and maintains database solutions for the Department of Veterans Affairs Loan Guaranty program, focusing on writing and optimizing complex PL/SQL stored procedures, packages, and SQL queries. Key functions include diagnosing and resolving data issues, designing efficient queries using advanced SQL features, and executing comprehensive test plans to ensure system reliability.

United States
$80K - $110K / year
Full TimeRemoteTeam 5,001-10,000

The role involves leading the design and implementation of robust, large-scale web scraping platforms using Python, while also mentoring junior developers and conducting code reviews. Responsibilities include developing sophisticated strategies to handle anti-bot countermeasures and optimizing scrapers for performance and stability.

PythonHTMLCSSJavaScriptHTTPPostgreSQLDockerKubernetesAWSScrapyPlaywright
United States

Connectivity Specialist

Airbnb

Airbnb is a community based on connection and belonging.

Backend Engineer4 days ago
Full TimeRemoteTeam 5,001-10,000Since 2007H1B Sponsor

This role involves providing specialist API support and acting as a subject matter expert for connectivity operations, including conducting analysis and technical troubleshooting to resolve complex issues. Responsibilities also include monitoring key metrics, serving as a technical point of contact for partners, and contributing to knowledge base improvements.

United States
$111K - $130K / year