NLP support with Huggingface tokenizers¶

This module contains the NLP support with Huggingface tokenizers implementation.

This is an implementation from Huggingface tokenizers RUST API.

Documentation¶

You can build the latest javadocs locally using the following command:

./gradlew javadoc

The javadocs output is built in the build/doc/javadoc folder.

You can pull the module from the central Maven repository by including the following dependency in your pom.xml file:

<dependency>
    <groupId>ai.djl.huggingface</groupId>
    <artifactId>tokenizers</artifactId>
    <version>0.17.0</version>
</dependency>