Glossary

batch data pipeline

A data pipeline that operates on batches of data. A batch is a set of data elements that are processed at the same time.

convolutional neural network (CNN, conv. net)

A neural network that has the convolution operation among its linear operations.

An early example is LeNet^[1]:

CNNs are based on discoveries about how brains work^[2]:

Illustration of "simple cells" by Hubel & Wiesel.

Visual neuron type	A related PyTorch module	Function
"Simple cell"	nn.Conv2d	Detects patterns
"Complex cell"	nn.MaxPool2d	Aggregates detections

Gradient Based Learning Applied to Document Recognition by LeCun et al. ↩︎
Receptive fields, binocular interaction and functional architecture in the cat's visual cortex by Hubel & Wiesel. ↩︎

data pipeline

A computational graph that's run in stages. Usually has a high degree of parallelism.

executor (data pipelines)

Logic that runs parallel operations across machines.

hard disk drive (HDD)

Hard disk drive.

Used in: Making Neural Networks Run Fast

machine learning (ML)

Using data to create logic.

Both statistics & machine learning study the use of observations to make inferences.

In my view, the key differences are:

The ML community leans more into procedural, algorithmic methods.
The ML community can place less necessity on theoretical understanding of methods, and more value on emperical validation.

Used in: The Alchemy of Semi-Supervision, Making Neural Networks Run Fast, Batch Data Pipeline Infrastructure

neural network

A statistical model that relates input random variables to output random variables via a sequence of linear & non-linear operations.

orchestrator (data pipelines)

Logic that resolves dependencies between stages and decides what runs when.

semantic segmentation

The computer vision task concerned with estimating $Y | X$ , where $X$ is a natural image and $Y$ is a per-pixel classification image.

Also known as image segmentation.

I think it's the most simply posed task in computer vision.

One might say: "isn't whole-image classification simpler?"
Consider: you could have many classes in an image, but with per-pixel classification, it's more feasible to have one unambiguous class per pixel.

Used in: The Alchemy of Semi-Supervision

solid state drive (SSD)

Solid state drive.

Used in: Making Neural Networks Run Fast

state of the art (SOTA)

The best known result for a task.

Used in: The Alchemy of Semi-Supervision

stream data pipeline

A data pipeline that operates on streams of data. A stream is a sequence of data elements that are not processed at the same time.

See also: data pipeline

task scheduler

Logic that assigns jobs to machines.

U-Net

An image-to-image model architecture introduced in a 2015 paper^[1].

I think U-Net can be "derived" in a sequence of simple steps.

Step 1. Premise

Say we want to create an image-to-image model using convolutions:

"F" - feature maps, "C" - convolution operations.

Step 2. Backbone

To learn useful representations, we might want to have a VGGNet-style backbone, that applies filters & downsamples repeatedly.

"F" - feature maps, "C" - convolution operations,
"D" - downsampling operations.

Step 3. Skip-connections

We ultimately want to produce an image output.

And residual connections are useful in neural networks (it's been shown empirically, and I think it makes sense theoretically, because they introduce redundancy).

We upsample with a reverse backbone, and wire it up with the existing backbone:

"F" - feature maps, "C" - convolution operations,
"D" - downsampling operations, "U" - upsampling operations.

And now this is the essential form of U-Net:

U-Net: Convolutional Networks for Biomedical Image Segmentation by Ronneberger, Fischer, and Brox ↩︎

Glossary

B

C

D

E

H

M

N

O

S

T

U