Scrapy Jobs

Project Title: Large-Scale Course Data Scraping from Udemy, Coursera, and YouTube Project Overview: I am looking for an experienced data scraping specialist who can collect a large dataset of online courses and tutorials from platforms such as Udemy, Coursera, and YouTube. The dataset should include both technical and non-technical tutorials. The objective is to build a structured dataset that can be used for research and analytics purposes. Project Requirements: The freelancer will scrape approximately 200,000 course/tutorial records from the following platforms: - Udemy - Coursera - YouTube The collected dataset should contain the following information for each record: 1. Course / Tutorial Title 2. Course URL / Video URL 3. Platform Name (Udemy / Coursera / YouTube) 4...

We need a robust yet lightweight script that can automatically pull business details from a publicly accessible government website. The information to capture will centre on business details, such as registration numbers, business name, registration date, address, etc. The workflow should: • Navigate every relevant section of the site (pagination, search filters, subsidiary pages). • Extract the required fields accurately • Export clean, structured data to CSV and JSON A Python solution leveraging requests/BeautifulSoup or Scrapy is preferred, but I’m open to other dependable stacks if they handle rate-limits, retries, and potential CAPTCHA gracefully. The script must be easy to rerun on demand, with clear instructions for environment setup and any dependencies...

Seeking a skilled web designer to create a clean, minimalistic website to showcase services. The website's primary function will be to highlight our offerings in a sleek, digestible format that engages visitors and inspires a sense of professionalism and quality. Features needed include: - A contact form to facilitate customer inquiries - Subscriber sign-up option for customers to receive updates and newsletters - A blog/news section to post updates and relevant content - An 'add listing' feature to display products effectively - Banner slide out et - Social Scroll It's crucial for the designer to have a keen eye for creating layouts that promote readability and user experience. Experience in minimalistic design can use template. Images & website examples will be...

# Python Developer (Scrapy / FastAPI / Docker) for Crawling Project We are looking for an experienced Python developer to build a structured web crawling system. This is not a large corporate project. It is a technically clean, well-defined system with a long-term perspective. We value clean architecture, stability, and understandable code. The system will process approximately 30,000 profiles per week and will be connected to an existing Laravel-based admin dashboard. --- ## Responsibilities * Develop a crawler using **Scrapy** * Build a small **FastAPI interface** (start / stop / status / statistics) * Integrate with **MariaDB** * Implement a media pipeline: download → local cache → upload to Wasabi (S3) * Implement hash-based change detection * Implement soft-delete logi...

I am looking for an experienced Web Scraping and Automation Developer to build an automated daily workflow for my business. I respond to over 100 government tenders a year and need to automate the discovery and document retrieval process. The Goal: Scrape 9 Australian Government tender websites daily for newly published tenders in two specific categories (UNSPSC 43000000 - IT, and 81000000 - Engineering/Research). Extract key details (Title, Agency, Closing Date, URL) and insert them into a centralized Google Sheet. Automatically download the associated Tender Documents and save them into uniquely named folders in my Google Drive. Periodically monitor these specific tenders for any newly published Addendums, and automatically download them to the respective Google Drive folder. The Website...

I have a list of roughly 1500 URLs—each coming from the same automotive website—that together cover the top 100 makes, models, grades and variants sold in Australia. I need every data point the site makes available for each of those vehicles, from the obvious specs such as year, make, model and variant right through to driveway prices, engine type, transmission, drive configuration, warranty details, fuel-economy figures, in-car technology features, seating layouts and any other attributes exposed on the page. The end goal is a clean, analysis-ready Excel workbook that lets me run market-wide comparisons, so consistency is critical: headings must be standardised, units normalised and categorical values written the same way across the entire sheet. I am happy for you to use P...

I need a reliable script that can pull live pricing details for car rentals from both the rental companies’ own sites and the big aggregator platforms in one pass. The goal is to feed it pickup / drop-off locations, dates, and driver age, then receive a clean CSV or JSON that lists vehicle class, daily and total price, currency, taxes & fees, and the URL it was scraped from. The scraper has to: • Navigate each target site automatically, including date pickers and location selectors. • Rotate user-agents / proxies or apply any other anti-bot tactics necessary to stay undetected. • Capture and log errors so a failed request never silently drops a row. • Be easy for me to rerun on demand—command-line or small web UI is fine, as long as setup is s...

I want to build a clean, well-structured dataset of restaurants throughout Portugal by scraping publicly available information on Google (Search / Maps). The core of the job is to capture three things for every venue you find: • complete contact details (name, address, phone, email or website if listed) • ratings plus review count pulled exactly as Google displays them • whether the place offers a fixed-price menu (“menu do dia”) and, if so, its price and what is included (possible 5 stages: Entrance, Plate, Drink, Dessert, Coffee) but it can be only 2 stages out of 5 (example: Plate+Drink) Please also tag each record with its district and municipality so the file can be filtered regionally. Deliverables 1. A single CSV or Excel file containing one row ...

I need you to collect 1,000 product images, titles, and links from Temu. Just these three items. I think this should be pretty straightforward. If you do a good job, I’ll hire you on a long-term basis, as I’ll need someone to help me collect product information from Temu on an ongoing basis.

I need a reliable developer to build a fully-automated system that gathers fresh, publicly listed email addresses from Google search results (pulled through a SERP API or an equivalent method) every single day, verifies them for validity, and delivers a clean CSV ready for my marketing campaigns. Here’s what I’m after: • Workflow 1. Submit a set of keywords or niche phrases. 2. Crawl the top Google result pages returned by a SERP service, extract any visible email addresses, and capture the source URL and page title. 3. Run each address through an SMTP-level verifier (ZeroBounce, NeverBounce, or an in-house Python verifier—whichever you prefer, as long as it returns status codes for valid, invalid, catch-all, disposable, and role accounts). 4. O...