1. Clicking the box programmatically – possible but inconsistent 2. Outsourcing the task to one of the many CAPTCHA-solving services (2Captcha etc) – better 3. Using a pool of reliable IP addresses so you don't encounter checkboxes or turnstiles – best I run a web scraping startup (https://simplescraper.io) and this is usually the approach. It has become more difficult, and I think a lot of the AI crawlers are... - Source: Hacker News / 4 days ago

Making my data extraction Saas (https://simplescraper.io) more LLM friendly. Markdown extraction, improved Google search, workflows - search for this terms, visit the first N links, summarize etc. Big demand for (or rather, expectation of) this lately. - Source: Hacker News / 7 months ago

Things are much easier for one-person startups these days—it's a gift. I remember building a todo app as my first SaaS project, and choosing something called Stormpath for authentication. It subsequently shut down, forcing me to do a last-minute migration from a hostel in Japan using Nitrous Cloud IDE (which also shut down). Just pain upon pain.[1] Now, you can just pick a full-stack cloud service and run with it.... - Source: Hacker News / 11 months ago

Simplescraper — Trigger your webhook after each operation. The free plan includes 100 cloud scrape credits. - Source: dev.to / over 1 year ago

I run Simplescraper (https://simplescraper.io). Started in 2020 and it's now profitable. > Have any recent trends affected your business? Not really. People like data as much as ever. As a one-person biz, the main dilemma remains how to juggle development, marketing and support. Reaching a point where the price of context-switching to customer support is becoming a little too high. But that's easily fixable and... - Source: Hacker News / about 2 years ago

This is where automation comes into play, transforming the data collection process from a chore into a streamlined operation. By utilizing web scraping tools, such as Automatio.ai, companies can automate the collection of vital information — such as competitor prices, market trends, or product availability — without lifting a finger. This data is then fed directly into Google Sheets, making it easy for teams to... - Source: dev.to / 7 months ago

Once your website monitoring is in place, automating online tasks and performance checks becomes the next logical step. Automatio, a no-code automation platform, makes it easy to manage web-based tasks without technical skills. Whether you need to monitor real-time metrics, test link functionality, or scrape reports, Automatio provides customizable bots that reduce manual work and ensure seamless operations. - Source: dev.to / 7 months ago

The combination of Google Sheets, Excel, AI tools, and automation platforms offers immense potential for web scraping. Whether extracting product prices, tracking leads, or performing SEO audits, each tool brings unique strengths. Bardeen.ai provides effortless no-code scraping workflows, while Automatio.ai enables seamless data integration with Google Sheets. Excel’s Power Query and macros offer advanced... - Source: dev.to / 7 months ago

A few years ago, we had a client and built a price-monitoring app for women's beauty products. They had multiple marketplaces, and like someone mentioned before, it was tricky because many products come in different sizes and EANs, and you need to be able to match them. We built a system for admins so they can match products from Site A with products from Site B. The scraping part was not that hard. We used our... - Source: Hacker News / 9 months ago

Https://automatio.co A no code web automation bot builder. It's in production, but still need to work on scaling the product, users. - Source: Hacker News / over 1 year ago