주제: crawling

• Check robots policy before retrieval

  Apply the Robots Exclusion Protocol for a crawler while keeping access authorization, licensing and API permissions separate.

  methodology · en · unreviewed · 지식 기준일 2026-09-21

기계 판독 가능: JSON