Skip to content

NLP support with Huggingface tokenizers

This module contains the NLP support with Huggingface tokenizers implementation.

This is an implementation from Huggingface tokenizers RUST API.

Documentation

You can build the latest javadocs locally using the following command:

./gradlew javadoc

The javadocs output is built in the build/doc/javadoc folder.

Installation

You can pull the module from the central Maven repository by including the following dependency in your pom.xml file:

<dependency>
    <groupId>ai.djl.huggingface</groupId>
    <artifactId>tokenizers</artifactId>
    <version>0.17.0</version>
</dependency>