Meta Deletes Commercial Clause Allowing AI Training Web Scraping
Meta has eliminated a controversial clause from its commercial terms of service following swift public pushback over broad automated data harvesting permissions. The language had authorized the social media parent company to extract and ingest digital content from linked external domains and web addresses. The policy reversal comes amid heightened sensitivity regarding how foundational artificial intelligence models source copyrighted editorial and enterprise text across the internet.
Article 53 of the European Union AI Act establishes obligations for providers of general-purpose AI models to maintain technical documentation and make it available to authorities and AI system providers.
Commercial Terms Revision and Web Scraping Controversy
TechCrunch noted that Meta recently updated its enterprise commercial terms of use to define broad content ingestion parameters. The draft legal provisions stipulated that the corporation could extract material from commercial URLs to train and optimize internal machine learning systems. Under the clause, any enterprise or news publisher maintaining a professional business profile risked having its proprietary published web pages crawled for commercial AI development.
Under Meta's revised Commercial Terms, contractual liability is capped at the greater of $100 or fees paid in the prior twelve months, and disputes involving US businesses must be resolved through individual arbitration.
Legal Scrutiny and Robots Exclusion Standards
Legal scholars including Cornell University information law professor James Grimmelmann warned that the contract clause introduced severe intellectual property complications. Specialists highlighted that the wording appeared to grant Meta sweeping authority to repurpose third-party copyrighted text without financial compensation or separate licensing contracts. Crucially, digital governance experts cautioned that the contractual phrasing could allow automated scrapers to override conventional web crawling restrictions specified in standard robots.txt exclusion files. Digital rights organizations noted that commercial contract terms should not circumvent established web protocol standards governing automated search engine indexing and bot access.
Meta Corporate Clarifications and Crawler Policies
Addressing the public controversy, a Meta spokesperson stated that the modified text was never designed to alter foundational web crawling behaviors. Corporate communications clarified that the consolidated rules were intended exclusively for advertising platforms and commercial utility products. The company reiterated that external website publishers retain full capability to block automated indexing bots by defining clear permissions within their server exclusion files. Andy Stone, Meta's vice president of communications, emphasized that crawler practices remain unchanged and dismissed claims of aggressive data collection as unfounded.
Across 107 prominent sites evaluated in a June 2026 robots.txt snapshot, 35 sites blocked at least one Meta AI crawler, with news publishers representing 11 of those blocks.
Broader Tech Industry Precedents and Training Consent
The dispute reflects widespread friction across the technology industry regarding automated training datasets for generative machine learning architectures. While xAI trains models on public microblog posts and Meta utilizes adult user submissions, both practices regularly proceed without explicit affirmative opt-in consent from creators. Although Meta secured a legal victory in a prominent copyright lawsuit last year, numerous parallel infringement claims against major AI developers remain active across federal courts. Similar consumer backlash occurred in July 2025 when file-sharing platform WeTransfer updated service contracts before retracting provisions that allowed algorithmic training on uploaded user files. The swift policy retractions highlight the growing regulatory and public resistance confronting tech enterprises that expand artificial intelligence training access without explicit agreements.
In June 2025, Judge Vince Chhabria ruled on a motion for summary judgment in Kadrey v. Meta, finding fair use for Meta's AI training activities.
In October 2024, Meta struck a multi-year AI deal with Reuters.
In October 2024, Twitter/X updated its Terms of Service to allow user posts to be used for AI training without an opt-out, but reversed the policy shortly after intense user backlash.