Scraper / Crawler for a website. 400 000 + pages.

Hi, I am a Professional Python Developer. I have scraped various websites. Let me describe what i understood about the job. 1. You want to scrape the link provided in the database to get reviews of customers. 2. The code should be automated, i.e. it should run automatically after each day. 3. The site stops bot to scrape so i need to bypass that. 4. Both Dynamodb and mysql are good choices. It operations that you will perform on database decides which one should be used. By the information given here, MySQL sound better. But we need to discuss about the operations on database before i can say MySQL. It won't cost you to just have a chat with me. I am very friendly and good person. I will be glad to help you with this. With Regards, Shubham Sharma

Hi I can implement a desktop application i C# that can run in windows. The data can be exported to Excel or mysql Thanks

Hello, I have read and understood what has to be done in this task. I have sound experience in performing such kind of work as I have done lot of project of similar kind in the past. I think I can assist you in getting this task done with more focus on quality and getting it delivered on time. Regards, Nidhi

Hello Kristijan, I can scrape the data, i have scrapped data from 5)+ wesbites, i have 9+ years of experience, please send me a message so we can discuss more. Thanks!

Hi employer. i am a expert in web scrapping using python. so if you give me this project, i will satisfy you surely. and i can show previous projects to you. please ping me kindly. thanks.

Hello! I can propose you to make a Python + Scrapy script, that will crawl reviews and save data to csv, json, mysql (on your choice). The script can be put on the cloud and set up to start once a day... It will also use proxies to avoid getting blocked.

I am an excellent web and software developer with over 7+ years experience. I want to use this skills to provide you good works. Please provide more information on your project and get it done. Thanks.

hi there hope you are good we can scrape 200000 pages data we have team work and we can do ti so just ping me.......thanks

I am a Professional Python Scraper I can do Scrapping of the Website with Proxy Enabled I am confident that i can do with high accuarcy Thanks in Advance

hello there , i have just checked the website with several requests so i did not see any blocking but maybe after a while can be of course. i will write a script in python with requests and bs4 module. but after testinng the code if there is any blocking i can go for selenium module with proxies. that can take 2 days or maybe shorter i cann not predict at the moment. firstly i have to write the test code and runn it for a while. i can directly connect the output to your mysql databank if u have it. if u dont we can create a table inn local with mysql or sqllite table. we can discuss about it this is the easiest part. so please let me know and we can discuss it further and then i can start to code. cheers Amadeus

Hello. I have checked your project description carefully. Web scrapping is my top skill and i can do it. I hope to work with you. Contact me please. Thank you.

Hi. I'm a programmer from Russia with strong skills in Python. This is not an automatic bid, I've read the description and the attachment. Proof: the website is called Kupujemprodajem. I'm good at web-scraping, browser automation and data extraction. I use Grab, Requests, Selenium and BeautifulSoup libraries for this purpose. The fastest solution for your case will be Requests + BeautifulSoup, but if the website has complex anti-scraping protection or complex Javascript, I will need to use Grab or Selenium. I performed complex web scraping project that concerned Amazon and Linkedin, AJAX request and passed anti-scraping tools. I worked a lot with Amazon and its APIs (Product and MWS) and even built scrapers for Amazon. The main problem with that was that Amazon blocked the scrapers if it received too many requests. I solved the problem with proxies or made less requests. About the database: I have basic knowledge of databases and SQL queries that helps me to create applications that manage and store data. Mostly I worked with PostgresQL and SQLite, but I'm also familiar with MySQL and Mongo. I'll be happy to answer your questions. With best regards, Sergey.

Hi there, You can choose 2 options, 1) chrome Extension 2) VB Script Advantage of VB Script is that it will run in background, but chrome Extension should have to be operated live. And daily 400,000 is not possible, as each page takes time to load, then only we can fetch data, if you want you can run the same software on different machine or same machine with different indexing. Lets chat to discuss this in details.

Greetings Kristijan python developer from Serbia here. I can provide you with scraping bot that you need. Zdravo Kristijane Pythonista iz Srbije, mogu da Vam dostavim bota kog zelite. Mladen

Hi Kristijan, We have extreme experience in Web Scraping. Have done a lot of work of Scraping using python. We use tools scrappy, selenium, scoop for scraping. We scraped data from many e-commerce websites and pushed it in excel and in a database. Also used selenium for creating Wassup bot. Ak

Hello, I can develop a Windows desktop application using C# for your project. It will be better and more stable for HTML changes or anti-bot measures. Thanks.

I can create a soft that will scrape info you need I can easily integrate it with sql, so results could be saved as csv files or imported direct to database I will create it with UBot Studio, you will get the SOFT only, EXE file To integrate proxy is not a problem but usual free proxy are not good, so you will need to buy proxy what I am offering you for $200: 1. Software, EXE file 2. 1 month free support/update 3. 1 month free us of my private proxies

"Hello Dear, Yes, I can work for you as per your project description with amazing exploration and experimental research abilities of mine organization is perfect for that you may require. My work fulfills as well as keep you returning consistently. It is because of the importance behind this field that I am confident & serious when taking these types of projects. Let me introduce with professionalism and skill that exists to supply the best. Regards"

Hello I am very professional in "Entering Types of Information - excel & Data Scraping". You can view employers' comments in this regard. I'm starting now and finishing it. Be sure you will be my regular customer. Send me a message. Maybe we agreed. Thank you for your attention.

I am doing web scrapping from past 6 months. I have scrapped many sites which includes IMDB websites and my college websites. I do this with beautiful soups.