Methodology
How the crawl works, and what it can't tell you.
Each entry's status comes from an automated attempt to read a public page. These are real crawl attempts, never human verification.
Rules the crawler follows
- It respects robots.txt.
- It reads public pages only: no logins, no forms, no paywalls.
- It does not bypass blocks, CAPTCHAs or rate limits.
Changes need an editor
A detected change never edits an entry by itself. A person must review the new page before updating policy observations. Our monitor's new fetch date never becomes the verification date for imported prices or rules.
What runs automatically
The API service checks for overdue sources on startup and hourly while it is running. Each approved publisher domain is due for another attempt after seven days. It discovers a public submission, guidelines, pricing or terms link and monitors that selected page, not every page on the website. Publishers sharing a domain share one source record.
When the service is stopped or asleep, checks wait until it runs again. The displayed dates, not a fixed schedule promise, are the record of what actually happened. Reliable unattended operation requires an always-running service or an external scheduled job.
All 296 distinct catalogue domains are in the fixed monitoring queue. Newly restored entries remain visible as “Not crawled yet” until an actual attempt completes; results update in the directory while the service runs.
Fetch success is not eligibility
A page loading tells you the page exists. It does not tell you the place accepts adult products, accepts yours, or will respond. We give no guarantees.
Status meanings
- Fetched, awaiting editor
- A public page was fetched. No human has judged it.
- Fetched, same text
- The text matched the previous fetch.
- Text changed
- The text differs from the last fetch. Editorial review is required before an entry changes.
- Crawler blocked
- robots rules or the server refused us. We stop there.
- Fetch error
- The attempt failed (network, timeout, server error).
- No policy page found
- No public submission or policy page was located.
What each entry shows
Each entry shows an imported policy record with its source-reported check date, reported price and link type, plus expandable reported rules, review notes and provenance. Our own monitor is a separate block with the latest attempt status, fetch or attempt date, source link when found, and captured excerpt when available. A failed check preserves the previous successful snapshot and labels its date separately. Historical imported observations are never presented as newly verified by List NSFW.
Catalogue data and licence
The baseline inventory contains 284 main listings and 13 restriction-reference listings. The baseline dataset was updated 1 October 2026 and imported 3 October 2026.
Data licence notice
The directory inventory, historical policy observations and adapted historical source notes include material from List NSFW AI. Public dataset: source export updated 1 October 2026; imported 3 October 2026.
This material is licensed under Creative Commons Attribution 4.0 International.
Changes: renamed and reordered categories, reformatted records, original List NSFW presentation, and separate independent source-monitoring results. Historical source observations are not independently verified by List NSFW. The source does not endorse List NSFW.
List NSFW uses its own design, descriptions, category names and ordering. The export is a baseline snapshot, not a live connection to the source website. Prices, acceptance classifications, review notes and source check dates are labelled as imported, not as our findings. Traffic rankings are not imported. A restriction-reference classification is historical source data and needs a current publisher-policy check. Listing availability is independent of whether our monitor has checked it.
Elsewhere: Hot Muses Universe
Last updated: 3 October 2026