On this page
THE DIRECT ANSWER
An XML sitemap lists preferred public URLs for discovery. Robots.txt manages crawler access. Neither replaces navigation, accurate publication settings, or proper protection for information that must remain private.
Keep the sitemap focused on intended URLs
List useful public pages in their preferred form and remove obsolete or redirected entries. For a bilingual library, ensure the language pages and alternatives agree with the actual site. Submit or reference the sitemap after it is available at the correct live address.
Do not use robots.txt as a privacy system
A disallow rule is not access control and does not reliably remove a URL from search results. Private information needs appropriate protection. When a page should not be indexed, plan the correct publishing mechanism rather than assuming that blocking crawling solves every visibility requirement.
Verify what is actually deployed
Open the live sitemap and inspect representative URLs. Check that the site's existing root robots configuration is not overwritten by a folder-only content package. A submitted sitemap is a discovery hint, not a promise that every listed page will be indexed.
An illustrative business scenario
An article package can place its own sitemap at /artikel/sitemap.xml while the main website keeps its existing sitemap. The owner can add the article sitemap to the established submission process without replacing unrelated site configuration.
A practical checklist
- List the preferred public URLs.
- Exclude obsolete sitemap entries.
- Preserve existing root configuration.
- Verify the live sitemap location.
Questions about this topic
Can robots.txt hide confidential documents?
No. Use appropriate access protection for confidential information rather than relying on crawler instructions.
Does submitting a sitemap guarantee indexing?
No. It helps communicate available URLs, but the search service still decides what to crawl and index.
Further reading
Plan your next website step.
Discuss your business, the visitor journey, and the work you need with Web Developer Jakarta. A short brief can help start a useful conversation.
Discuss Your Website ↗