This is an integration environment. Data may not be preserved.
Skip To Content
Toggle navigation
Packages
Upload
Statistics
Documentation
Downloads
Blog
Sign in
Advanced search filters
Frameworks
Include compatible frameworks
Framework Filter Mode
ALL
ANY
.NET
net8.0
net7.0
net6.0
net5.0
.NET Core
netcoreapp3.1
netcoreapp3.0
netcoreapp2.2
netcoreapp2.1
netcoreapp2.0
netcoreapp1.1
netcoreapp1.0
.NET Standard
netstandard2.1
netstandard2.0
netstandard1.6
netstandard1.5
netstandard1.4
netstandard1.3
netstandard1.2
netstandard1.1
netstandard1.0
.NET Framework
net481
net48
net472
net471
net47
net462
net461
net46
net452
net451
net45
net40
net35
net30
net20
Package type
All types
Dependency
.NET tool
Template
Options
Include prerelease
9 packages returned for Tags:"tokenizer"
Sort by
Relevance
Downloads
Recently updated
Tokenizer
by:
flipbit
.NET Framework 3.5
393 total downloads
last updated
10/30/2014
Latest version:
1.0.1
tokenizer
tokens
pattern
matching
Tokenizer extracts structed information from blocks of text and reflects them onto .NET objects
OpenNLP.
NET
by:
sergey_tihon
.NET Framework
762 total downloads
last updated
2/13/2015
Latest version:
1.6.0-rc2
NLP
OpenNLP
Tokenizer
Categorizer
Chunker
Parser
POS
NER
The Apache OpenNLP library is a machine learning based toolkit for the processing of natural language text. It supports the most common NLP tasks, such as tokenization, sentence segmentation, part-of-speech...
More information
Conekta.
Xamarin
by:
bscheiman
.NET 6.0
353 total downloads
last updated
3/1/2015
Latest version:
1.0.6
boris
scheiman
bscheiman
conekta
xamarin
tokenizer
Tokenizador (Xamarin) para Conekta. Necesitas tener alguna libreria de servidor para usar el token.
VBF.
Compilers.
Scanners
by:
Ninputer
.NET Framework
215 total downloads
last updated
3/26/2015
Latest version:
1.0.1
VBF
Compiler
scanner
lexer
lex
tokenizer
VBF.Compilers.Scanners is a scanner builder. It contains a regular expression to DFA engine, can generate high performance scanners for unicode source text.
owl
by:
SplittyDev
346 total downloads
last updated
10/21/2014
Latest version:
1.0.5407.37498
html
preprocessor
compiler
lexer
parser
scanner
tokenizer
owl
A html preprocessor which allows you to write html code using a beautiful syntax
Stringes
by:
Berkin
.NET Framework 4.5
764 total downloads
last updated
4/5/2015
Latest version:
1.4.3
stringes
tokenizer
token
lexer
string
substring
parser
ast
syntax
grammar
More tags
The Stringe is a wrapper for the .NET String object that tracks line, column, offset, and other metadata for substrings.
Virastyar
by:
senobari
.NET Framework
363 total downloads
last updated
12/11/2011
Latest version:
2.0.0
spell-checker
transliterator
unicode
farsi
persian
NLP
language
tokenizer
datetime
lemmatizer
More tags
A Farsi (Persian) language checking and NLP library. This package includes both library and it's required data files.
Virastyar.
Lib
by:
senobari
.NET Framework
208 total downloads
last updated
12/11/2011
Latest version:
2.0.0
spell-checker
transliterator
unicode
farsi
persian
NLP
language
tokenizer
datetime
lemmatizer
More tags
A Farsi (Persian) language checking and NLP library.
Virastyar.
Data
by:
senobari
253 total downloads
last updated
12/11/2011
Latest version:
2.0.0
spell-checker
transliterator
unicode
farsi
persian
NLP
language
tokenizer
datetime
lemmatizer
More tags
Required data files for Virastyar library.