This is an integration environment. Data may not be preserved.
Skip To Content
Toggle navigation
Packages
Upload
Statistics
Documentation
Downloads
Blog
Sign in
Advanced search filters
Frameworks
Include compatible frameworks
Framework Filter Mode
ALL
ANY
.NET
net8.0
net7.0
net6.0
net5.0
.NET Core
netcoreapp3.1
netcoreapp3.0
netcoreapp2.2
netcoreapp2.1
netcoreapp2.0
netcoreapp1.1
netcoreapp1.0
.NET Standard
netstandard2.1
netstandard2.0
netstandard1.6
netstandard1.5
netstandard1.4
netstandard1.3
netstandard1.2
netstandard1.1
netstandard1.0
.NET Framework
net481
net48
net472
net471
net47
net462
net461
net46
net452
net451
net45
net40
net35
net30
net20
Package type
All types
Dependency
.NET tool
Template
Options
Include prerelease
20 packages returned for Tags:"crawler"
Sort by
Relevance
Downloads
Recently updated
HtmlAgilityPack
by:
darthobiwan
.NET Framework 2.0
693,435 total downloads
last updated
9/16/2014
Latest version:
1.4.9
HTML
parse
crawler
spider
linq
This is an agile HTML parser that builds a read/write DOM and supports plain XPATH or XSLT (you actually don't HAVE to understand XPATH nor XSLT to use it, don't worry...). It is a .NET code library that allows...
More information
Abot
by:
sjdirect
.NET Framework 4.0
13,568 total downloads
last updated
4/4/2015
Latest version:
1.5.0.93
crawler
robot
spider
Abot is an open source C# web crawler built for speed and flexibility. It takes care of the low level plumbing (multithreading, http requests, scheduling, link parsing, etc..). You just register for events to...
More information
SkyScraper
by:
JonCanning
.NET Framework 4.5
914 total downloads
last updated
6/14/2013
Latest version:
1.0.45
spider
crawler
scraper
bot
Web scraper / crawler / spider. Supports robots protocol and user agent.
RobotRules
by:
vodevil
.NET Framework 4.5
405 total downloads
last updated
6/10/2014
Latest version:
1.5.2.4
Robots.txt
robot
crawler
web
A Simple and Powerful Library to Deal with Web Robots Control Strategy.
MisterHex.
WebCrawling
by:
misterhex
.NET Framework 4.5
407 total downloads
last updated
4/2/2013
Latest version:
1.0.0.5
crawler
rx
crawling
spider
Simple web crawler that return IObservable using Reactive Extension(Rx) and async await.
NCrawler
by:
codevision
.NET Framework 4.5
376 total downloads
last updated
1/2/2015
Latest version:
3.0.1
NCrawler
crawler
spider
robot
Crawler and scrapping framework which is written in C#
Tenteikura
by:
bolthar
.NET Framework 4.0
137 total downloads
last updated
11/21/2012
Latest version:
1.0.0
Crawler
Web
A minimal multithreaded web crawler
NCrawler.
EntityFramework
by:
codevision
.NET Framework 4.5
236 total downloads
last updated
1/2/2015
Latest version:
3.0.2
NCrawler
crawler
spider
robot
Provides storing crawler data using EF for NCrawler
NCrawler.
HtmlProcessor
by:
codevision
.NET Framework 4.5
218 total downloads
last updated
1/1/2015
Latest version:
3.0.0
NCrawler
crawler
spider
robot
HTML processing of crawled data for NCrawler
totalrecall
by:
brandonc
.NET Framework
139 total downloads
last updated
6/8/2011
Latest version:
0.2.3
search
msbuild
crawler
lucene
Crawl and index your (static) asp.net website for searching using an MSBuild target. Simple search query interface. Uses Lucene.Net and NCrawler.
NCrawler.
EsentServices
by:
codevision
.NET Framework 4.5
124 total downloads
last updated
1/1/2015
Latest version:
3.0.0
NCrawler
crawler
spider
robot
This is ESENT storage providers for the NCrawler
NCrawler.
FileStorageServices
by:
codevision
.NET Framework 4.5
135 total downloads
last updated
1/1/2015
Latest version:
3.0.0
NCrawler
crawler
spider
robot
Storage in file system for the NCrawler
DataExtractingSDK
by:
DevRain
.NET Framework 4.0
341 total downloads
last updated
3/30/2012
Latest version:
1.0.1
data
extracting
html
processing
crawler
mining
Library allows you to extract different data from text or web sources, analyze text content, extract different information and write your own data mining & extracting applications and services.
scSearchContrib.
Searcher
by:
sitecorian
.NET Framework 3.5
282 total downloads
last updated
5/10/2013
Latest version:
2.1.0.1
sitecore
search
lucene
advanced
database
crawler
The searching/querying component of the Sitecore Search Contrib project
SocialSense
by:
fakeezz
.NET Framework 4.0
278 total downloads
last updated
8/31/2013
Latest version:
1.2.0
social
crawler
twitter
facebook
google+
blog
Crawler for main social networks
scSearchContrib.
Crawler
by:
sitecorian
.NET Framework 3.5
252 total downloads
last updated
5/7/2013
Latest version:
2.1.0
sitecore
search
lucene
advanced
database
crawler
The crawling/indexing component of the Sitecore Search Contrib project
Spidy
by:
Taha
.NET Framework 4.0
141 total downloads
last updated
9/24/2012
Latest version:
1.0.0
FSharp
Web
Crawler
Spider
Webbot
Bot
FSharp Web Crawler
CrawlerLib.
Engine.
TestHelper
by:
Crawler-Lib
.NET Framework
863 total downloads
last updated
3/7/2015
Latest version:
2.3.5544.21265
crawler-lib
test
helper
nunit
mstest
unit-test
crawler
spider
bot
scraper
More tags
The Crawler-Lib Engine Test Helper simplifies the test of tasks. It can be used to develop unit tests and integration tests for tasks.
dcsoup
by:
matarillo
.NET Framework 4.5
311 total downloads
last updated
3/21/2015
Latest version:
0.2.0
HTML
jsoup
jquery
parse
parser
parsing
crawl
crawler
crawling
scrape
More tags
dcsoup is a .NET library for working with real-world HTML. It provides a very convenient API for extracting and manipulating data, using the best of DOM, CSS, and jquery-like methods. This library is basically...
More information
CrawlerLib.
Engine
by:
Crawler-Lib
.NET Framework
1,386 total downloads
last updated
3/7/2015
Latest version:
2.3.5544.21265
crawler-lib
task
processor
workflow
action
async
await
crawler
spider
bot
More tags
The Crawler-Lib Engine is a general purpose workflow enabled task processor. It has evolved from a web crawler over data mining and information retrieval. It is throughput optimized and can perform thousands of...
More information