HuggingFace Transformer.js

Finally, I am able to talk about the huggingface transformer.js project. This project uses ONNX to make it possible to run HuggingFace models in the browser.

Just calling the API is boring, there is a react translation tutorial which is the best way to get started.

Download the model: in the model, they used a worker.js to download the model. the progress_callback is used to show the progress of the download.

this.instance ??= pipeline(this.task, this.model, { progress_callback });

The pipeline function is used to create a pipeline for the model. You can add parameter dtype: "fp32" for higher precision, but the file will be bigger.

device: "webgpu" is used to run the model on the GPU.

the progress_callback will return the progress event. The structure is like this.

{
    status: "initiate" | "progress" | "done" | "ready",
    data: {
        file: string, // file name
        progress: number, // progress percentage
        ...
    }
}

The following code shows how to stream the result. If you are using typescript, you may get errors, maybe the type definition is wrong.

// Listen for messages from the main thread
self.addEventListener('message', async (event) => {
  // Retrieve the translation pipeline. When called for the first time,
  // this will load the pipeline and save it for future use.
  const translator = await MyTranslationPipeline.getInstance(x => {
      // We also add a progress callback to the pipeline so that we can
      // track model loading.
      self.postMessage(x);
  });

  // Capture partial output as it streams from the pipeline
  const streamer = new TextStreamer(translator.tokenizer, {
      skip_prompt: true,
      skip_special_tokens: true,
      callback_function: function (text) {
          self.postMessage({
              status: 'update',
              output: text
          });
      }
  });

  // Actually perform the translation
  const output = await translator(event.data.text, {
      tgt_lang: event.data.tgt_lang,
      src_lang: event.data.src_lang,

      // Allows for partial output to be captured
      streamer,
  });

  // Send the output back to the main thread
  self.postMessage({
      status: 'complete',
      output,
  });
});

I can download the model and load locally.

import { env } from '@huggingface/transformers';
env.allowRemoteModels = false;
env.localModelPath = '/path/to/local/models/';

Most important is the pipelines. It is a list of pipelines that defines what it can do and what it cannot do.

pipeline	category	example models	input	output
TextClassificationPipeline	sentiment-analysis	Xenova/distilbert-base-uncased-finetuned-sst-2-english	’I love transformers!’	[]
	text-classification	Xenova/toxic-bert	’I hate you!’	[,… ]
TokenClassificationPipeline	token-classification	Xenova/bert-base-NER	’My name is Sarah and I live in London’	[,]
QuestionAnsweringPipeline	question-answering	Xenova/distilbert-base-uncased-distilled-squad	const question = ‘Who was Jim Henson?’; const context = ‘Jim Henson was a nice puppet.’;	answer: "a nice puppet", score: 0.5768911502526741
FillMaskPipeline	fill-mask	Xenova/bert-base-cased	’The goal of life is [MASK].’	[, , , , ]
Text2TextGenerationPipeline	text2text-generation	Xenova/LaMini-Flan-T5-783M	’how can I become more healthy?’,	[]
SummarizationPipeline	summarization	Xenova/distilbart-cnn-6-6	text,	[]
TranslationPipeline	translation	Xenova/nllb-200-distilled-600M	’जीवन एक चॉकलेट बॉक्स की तरह है।’,	[]
TextGenerationPipeline	text-generation	Xenova/distilgpt2	I enjoy walking with my cute dog,	[]
ZeroShotClassificationPipeline	’zero-shot-classification	Xenova/mobilebert-uncased-mnli	const text = ‘Last week I upgraded my iOS version and ever since then my phone has been overheating whenever I use your app.’; const labels = [ ‘mobile’, ‘billing’, ‘website’, ‘account access’ ];	sequence: 'Last week I upgraded my iOS version and ever since then my phone has been overheating whenever I use your app.', labels: [ 'mobile', 'website', 'billing', 'account access' ], scores: [ 0.5562091040482018, 0.1843621307860853, 0.13942646639336376, 0.12000229877234923 ]
FeatureExtractionPipeline	feature-extraction	Xenova/bert-base-uncased	This is a simple test.	Tensor
ImageFeatureExtractionPipeline	image-feature-extraction	Xenova/vit-base-patch16-224-in21k	image_url	Tensor
AudioClassificationPipeline	audio-classification	’Xenova/wav2vec2-large-xlsr-53-gender-recognition-librispeech’	audio_url	[, ]
ZeroShotAudioClassificationPipeline	zero-shot-audio-classification	Xenova/clap-htsat-unfused	audio_url,	[,]
AutomaticSpeechRecognitionPipeline	automatic-speech-recognition	Xenova/whisper-tiny.en	audio_url	text: 'Hello, how are you?'
ImageToTextPipeline	image-to-text	Xenova/blip2-opt-2.7b	image_url	[]
ImageClassificationPipeline	image-classification	’Xenova/vit-base-patch16-224	image_url	[]
ImageSegmentationPipeline	image-segmentation	Xenova/detr-resnet-50-panoptic	image_url	[{ label: ‘cat’, score: 0.9999998807907104, mask: RawImage }]
BackgroundRemovalPipeline	background-removal	’Xenova/modnet’	image_url	[RawImage ]
ZeroShotImageClassificationPipeline	zero-shot-image-classification	Xenova/clip-vit-base-patch32.	url, [‘tiger’, ‘horse’, ‘dog’]	[, , ]
ObjectDetectionPipeline	object-detection	Xenova/detr-resnet-50	img,	[{score: 0.9976370930671692, label: “remote”, box: },… ]
ZeroShotObjectDetectionPipeline	Zero-shot object detection	Xenova/owlvit-base-patch32	img,	[{ score: 0.9999998807907104, label: ‘remote’, box: }, … ]
DocumentQuestionAnsweringPipeline	document-question-answering	image, ‘What is the invoice number?’	[]
TextToAudioPipeline	text-to-speech	Xenova/speecht5_tts	’Hello, my dog is cute’,	RawAudio

• ImageToImagePipeline： enlarger an image
• DepthEstimationPipeline: make depth graph