Robots.txt Tester — Validate Crawl Rules

Want to create short URLs? Sign up for free to get started.

About Robots.txt Tester

What is the Robots.txt Tester?

Validate whether a specific URL is allowed or blocked for a given crawler. Use it to confirm SEO directives before launch.

How It Works

Paste a robots.txt file or fetch one from a domain.
Enter a user-agent and URL or path to test.
See the matched rule and allow/disallow result.

Great For

SEO QA: Verify crawl rules for key pages.
Migrations: Ensure bots can reach updated URLs.
Troubleshooting: Identify blocking rules quickly.

After testing robots rules, you may also want Robots.txt Generator, Sitemap Finder & Checker, and Sitemap Validator.

Related Tools

More tools you may find useful.

Add Rainbow Filter to Image Add Text to Photo AVIF Converter Base64 to ASCII Converter Base64 to Audio Converter Base64 to File Converter Base64 to Hex Converter Base64 to Image Converter Base64 to PDF Converter Base64 to Text Converter Base64 to Video Converter Bulk Image Resizer Canonical URL Checker Change Horizontal Image to Vertical Change Vertical Image to Horizontal CMYK To Pantone Color Converter (HEX/RGB/HSL) Color Palette Generator Color Picker From Image Conversion Rate Calculator

Recent Blog Posts

Introducing the T.LY Android App

T.LY now has a native Android app for phones and tablets. It brings the work people already do in T.LY to Android: create short links, manage campaign links, check clicks, watch QR scans, and make cha...

5 MIN READ

Split QR Code Phishing: Why a Clean PDF Can Still Be Dangerous

A phishing email lands in an employee's inbox with a PDF attached. It says their compensation has changed and asks them to scan a QR code to read the details. The PDF looks clean. There is no clickabl...

8 MIN READ

Introducing BeforeTap for iOS

On a phone, every link asks you to tap before you know exactly where it leads. A short URL can hide its destination. A QR code hides the URL completely. Even a normal-looking link can pass through sev...

4 MIN READ

How to Check if a QR Code Is Safe Before You Open It

QR codes are everywhere because they work. I use them for menus, signs, product pages, Wi-Fi, and plenty of T.LY campaigns. The catch is simple: you cannot read a QR code by looking at it. The destina...

9 MIN READ

How to Track Influencer Marketing With Short Links and QR Codes

Influencer marketing is hard to track when everyone shares the same link. One creator posts on TikTok. Another adds a story link. Someone else hands out promo cards at an event. If they all use the sa...

2 MIN READ

Introducing the T.LY iOS App

T.LY now has a native iOS app for iPhone and iPad. It is built for the work people already do in T.LY: create short links, manage campaign links, check clicks, watch QR scans, and make changes when so...

4 MIN READ

A Free QR Code Scanner & Creator App for iPhone and Android

Most QR scanner apps in the App Store and Google Play do the same frustrating thing: they take a tool that should be free and wall it off behind ads, sign-up screens, or a “pro” subscription. Want to...

2 MIN READ

T.LY Link Shortener Extension Now Supports Safari

The T.LY Link Shortener Extension now supports Safari. If Safari is your daily browser, you can create a T.LY short link for the page you are already viewing without opening another tab. Click the too...

1 MIN READ

Dashboard and Stats Refresh

T.LY analytics recently got a cleaner dashboard and stats page. This update was not just about making the screens look newer. The goal was to make link performance easier to understand when you have a...

8 MIN READ

IPv4 vs IPv6: What Is the Difference?

IPv4 and IPv6 are two versions of IP addresses. Both help devices communicate across networks. The main difference is address space. IPv4 is older and shorter. IPv6 is newer and supports a much larger...

5 MIN READ

How to Track QR Code Campaigns Before You Print

The best time to set up QR code tracking is before you print. Not after the flyers arrive. Not after the menu is laminated. Not after the conference badges are shipped. Before. Once a QR code is print...

8 MIN READ

How to Choose a Short Branded Domain

A short branded domain is the domain you use for your short links. Instead of sending people through a generic short URL, you use something tied to your brand, campaign, product, or organization. For...

9 MIN READ