tsllbn RobotsTxt by: tsllbn .NET Framework 1,004 total downloads last updated 2/19/2014 Latest version: 2014.2.19 Robots.Txt Parser Parsing Library A robots.txt parser for .NET Supports ; - Allow directives. - Crawl-delay directives. - Sitemap declarations. - * and $ wildcards.