Packages

  • package root
    Definition Classes
    root
  • package lamp

    Lamp provides utilities to build state of the art machine learning applications

    Lamp provides utilities to build state of the art machine learning applications

    Overview

    Notable types and packages:

    • lamp.STen is a memory managed wrapper around aten.ATen, an off the heap, native n-dimensionl array backed by libtorch.
    • lamp.autograd implements reverse mode automatic differentiation.
    • lamp.nn contains neural network building blocks, see e.g. lamp.nn.Linear.
    • lamp.data.IOLoops implements a training loop and other data related abstractions.
    • lamp.knn implements k-nearest neighbor search on the CPU and GPU
    • lamp.umap.Umap implements the UMAP dimension reduction algorithm
    • lamp.onnx implements serialization of computation graphs into ONNX format
    • lamp.io contains CSV and NPY readers
    How to get data into lamp

    Use one of the file readers in lamp.io or one of the factories in lamp.STen$.

    How to define a custom neural network layer

    See the documentation on lamp.nn.GenericModule

    How to compose neural network layers

    See the documentation on lamp.nn

    How to train models

    See the training loops in lamp.data.IOLoops

    Definition Classes
    root
  • package data
    Definition Classes
    lamp
  • package bytesegmentencoding

    Greedy contraction of consecutive n-grams

    Greedy contraction of consecutive n-grams

    Definition Classes
    data
  • ByteSegmentCodec
  • ByteSegmentCodecFactory
  • package distributed
    Definition Classes
    data
  • package schemas
    Definition Classes
    data
p

lamp.data

bytesegmentencoding

package bytesegmentencoding

Greedy contraction of consecutive n-grams

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. bytesegmentencoding
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Type Members

  1. case class ByteSegmentCodec(trained: Vector[(Vector[Byte], Char)], unknownToken: Char, unknownByte: Byte) extends Codec with Product with Serializable
  2. case class ByteSegmentCodecFactory(vocabularyMin: Char, vocabularyMax: Char, maxMergedSegmentLength: Int, unknownToken: Char, unknownByte: Byte) extends CodecFactory[ByteSegmentCodec] with Product with Serializable

Value Members

  1. def decode(encoded: Array[Char], encoding: Vector[(Vector[Byte], Char)], unknown: Byte): Array[Byte]
  2. def encode(corpus: Array[Byte], encoding: Vector[(Vector[Byte], Char)], unknownToken: Char): Array[Char]
  3. def readEncodingFromFile(file: File): ByteSegmentEncoding
  4. def saveEncodingToFile(file: File, encoding: Vector[(Vector[Byte], Char)], unknownToken: Char, unknownByte: Byte): Unit
  5. def train(corpus: Array[Byte], vocabularyMin: Char, vocabularyMax: Char, maxMergedSegmentLength: Int): Vector[(Vector[Byte], Char)]

    Trains BPE encoding

    Trains BPE encoding

    Char here is used as unsigned 16 bit integer

Inherited from AnyRef

Inherited from Any

Ungrouped