CommonCrawl Reviews and details

Social recommendations and mentions

• Writers' Call for OTW and Ao3 Transparency, Clarity, and Stance on A​.​I.

  There are programs that go onto websites and basically open every page on the website, downloading all or some of the text, and storing them. This process is known as "scraping". When people talk about it in the context of AI, they're referring to https://commoncrawl.org/. - Source: Reddit / 17 days ago

• ChatGPT Data Breach BreakDown - Why it Should be a Concern for Everyone!

  The dataset they use is the common crawl database https://commoncrawl.org/ you can just ask ChatGPT and it will tell you. - Source: Reddit / 23 days ago

• Is ChatGPT a reliable source for accurately debunking conspiracies, urban legends and myths?

  It can do the opposite of what you want perfectly if you know what you're trying to prove in great details. ChatGPT is very, very good at making plausible statements in accordance with what they feed it in Common Crawl, and various sources like that on the internet. - Source: Reddit / 28 days ago

• Europe to ChatGPT: Disclose your sources — Proposed legislation requires developers to list copyright material used in generative AI tools

  They aren’t just scraping the top ten websites or some shit. It’s a deep crawl of the entire internet. Petabytes of data. One of the main sources a lot of these LLMs use is this: https://commoncrawl.org. - Source: Reddit / about 1 month ago

• The Age of Giant AI Models is Over

  Interesting, thanks. I was just basing my claim off the fact that GPT uses Common Crawl (which incidentally is hosted on AWS). However, I don't know how much of the entire internet is actually included in Common Crawl. - Source: Reddit / about 1 month ago

• New UK Framework AI Rules Unveiled

  Not even "effectively". For a start, it's got nothing to do with Google search results (the main dataset is the open data Common Crawl which accounted for ~80% of the data used, along with WebText2, Wikipedia and other sources). - Source: Reddit / 2 months ago

• Preparing for the worst outcome for Internet Archive

  Yes, and OP and everyone else can perhaps focus on backup on Common Crawl etc. - Source: Reddit / 2 months ago

• Show HN: Ichido, search engine that tags sites using Google and Cloudflare

  What about Common Crawl? https://commoncrawl.org/. - Source: Hacker News / 3 months ago

• AI Is Useful for Capitalists but Probably Terrible for Anyone Else

  Yeah I posted this in another thread but your claim is simply untrue. There is little to no need for anyone to scrape the web themselves, CommonCrawl does it for free and provides access to all that data for free, they are a charity similar to The Internet Archive: https://commoncrawl.org/ As for using that data to train a model, I've done it at my own company and it's not that expensive, certainly not the... - Source: Hacker News / 3 months ago

• AI Is Useful for Capitalists but Probably Terrible for Anyone Else

  >There are only a handful of companies with enough data to be able to train artificial intelligence algorithms. https://commoncrawl.org/. - Source: Hacker News / 3 months ago

• chatgpt protecting evil corporate interest

  No-one paid to have it tweaked. It predicts the most probable sequence of word tokens based on your input (and its previous output if there was one). Its understanding of how similar words are to each other (word embeddings) was trained based on predicting the next word in a sequence, mainly from a dataset called of webpages called CommonCrawl (https://commoncrawl.org/). It doesn't "know" anything, it just has a... - Source: Reddit / 3 months ago

• Improving Search Quality for Non-English Queries with Fine-tuned Multilingual CLIP Models

  Open AI then co-trained the two encoders with the multilingual laion5b dataset, which contains 5.85 billion image-text pairs: 2.2 billion of these pairs are labelled in 100+ non-English languages, with the rest in English or containing text that can’t be nailed down to any one language (like place names or other proper nouns). These are taken from a sampling of images and their HTML alt-text in the Common Crawl... - Source: dev.to / 4 months ago

• OpenAI making it's next move: "AI should be regulated"

  Common Crawl is a good one to start off playing with. It's huge, covers a long span of time, and has both full and plaintext only versions. https://commoncrawl.org/. - Source: Reddit / 4 months ago

• Usurping collective Internet knowledge and presenting it as it's own

  In any case, this is part of the training data, I think: https://commoncrawl.org/. - Source: Reddit / 4 months ago

• Ask HN: I want to train a LM on my home country's dialect, how can I do it?

  If all you want is a LM and it doesn't need to be trained by you or run on infrastructure you control, you could try to see whether ChatGPT already understands well enough. A Tunisian friend of mine told me that he asked it to tell a joke in Tunisian Arabic and it worked, only the joke wasn't funny. If you want or need to train on your own data, social media is a good bet for colloquial language. You could try... - Source: Hacker News / 4 months ago

• When will companies release clones resembling ChatGPT? What time interval? What are the difficulties and challenges?

  Datasets are fairly easy to compile. 60% of GPT is from https://commoncrawl.org/ and using the. - Source: Reddit / 4 months ago

• How I archived 100 million PDF documents... (Part 1)

  The goal of the prototype was simple. I "just" wanted to download 10.000 documents to understand how hard it is to collect and kind of archive them. The immediate problem was that I didn't know where can I get links for this many files. Sitemaps can be useful in similar scenarios. However, there are a couple of reasons why in this case they are not really a viable solution. Most of the time it doesn't contain... - Source: dev.to / 5 months ago

• ChatGPT, draw a human in Python/OpenCV. Please draw detailed human. No, draw an attractive and detailed human. Here it is!

  AFAIK they used Common Crawl data as their training data. - Source: Reddit / 5 months ago

• Improving Search Quality for Non-English Queries with Fine-tuned Multilingual CLIP Models

  Open AI then co-trained the two encoders with the multilingual laion5b dataset, which contains 5.85 billion image-text pairs: 2.2 billion of these pairs are labelled in 100+ non-English languages, with the rest in English or containing text that can’t be nailed down to any one language (like place names or other proper nouns). These are taken from a sampling of images and their HTML alt-text in the Common Crawl... - Source: dev.to / 5 months ago

• ArtStation makes statement saying they'll allow AI art on the platform, and will give artists the option to opt-in or opt-out of having their artwork used in AI research

  And StabilityAI didn't even do their own web scraping, they used the Common Crawl data. - Source: Reddit / 5 months ago

• Looking for/if source images sampled by sd, mj, and other GANs is able to be tracked and sourced

  So a previous user got heated with some of my responses I guess and referenced that https://laion.ai/ uses https://commoncrawl.org/ which is a bot that scrapes data for the entire internet (petabites of data). - Source: Reddit / 6 months ago