NLP support with Huggingface tokenizers¶
This module contains the NLP support with Huggingface tokenizers implementation.
This is an implementation from Huggingface tokenizers RUST API.
Documentation¶
You can build the latest javadocs locally using the following command:
./gradlew javadoc
The javadocs output is built in the build/doc/javadoc
folder.
Installation¶
You can pull the module from the central Maven repository by including the following dependency in your pom.xml
file:
<dependency>
<groupId>ai.djl.huggingface</groupId>
<artifactId>tokenizers</artifactId>
<version>0.17.0</version>
</dependency>