Images and video are among the most common types of unstructured data in the world. Data Science — specifically Computer Vision and Deep Learning — enables machines to interpret, classify, and understand visual information at scale.
By the end of this chapter, you will be able to:
The following diagram maps the core computer vision tasks — from basic classification and detection to modern multimodal retrieval and generative modeling.
Loading diagram...
What it does: The AI looks at a picture and says what it is (like "this is a cat" or "this is a car").
What it does: The AI finds specific things in a picture, draws a box around them, and labels each one.
What it does: The AI colors in the exact shape of objects in a picture — like perfectly marking which pixels belong to what object.
What it does: The AI watches a video, follows where objects are moving, and understands what action is happening.
What it does: Combines all the vision techniques above to help a car "see" the road and drive itself safely.
Loading diagram...
What it does: The AI creates new pictures from text descriptions (like typing "a sunset over mountains" and the AI creates that image).
What this code does: Creates a new image based on a text description using AI.
Loading code block...
Output:
Image file saved as landscape.png — a beautiful oil painting of mountains and landscapes generated by AI based on your text description.
While CNNs were the gold standard for years, modern architectures have expanded the toolkit:
| Architecture | When to Use | Examples | Strengths |
|---|---|---|---|
| CNNs | Image classification, detection | ResNet, EfficientNet | Efficient, good for structured grids |
| Vision Transformers (ViTs) | Large-scale, few-shot learning | ViT, DeiT, BEiT | Better generalization, global context |
| Multimodal Models | Image + text understanding | CLIP, LLaVA, GPT-4o | Natural language prompting |
| Diffusion Models | Image generation, inpainting | Stable Diffusion, DALL-E 3 | High-quality image synthesis |
| Hybrid Architectures | Balanced efficiency & accuracy | ConvNeXt, MAE | Combines CNN speed with ViT breadth |
Traditional Convolutional Neural Networks (CNNs) dominated computer vision for a decade. Today's landscape is more diverse, with foundational models capable of zero-shot understanding.
Vision Transformers adapt the Transformer architecture (originally for NLP) to images by splitting them into patches and processing them like tokens.
What this code does: Uses a Google-pre-trained Vision Transformer (ViT) to preprocess and classify a dog image.
Loading code block...
Output:
Predicted class: golden retriever
Foundation models like CLIP and LLaVA understand both images and natural language, enabling "zero-shot" classification where the model can identify objects it wasn't explicitly trained on.
What this code does: Leverages an OpenAI CLIP model to perform zero-shot classification against a list of candidate labels.
Loading code block...
Output:
a dog: 94.32%
a cat: 4.21%
a car: 1.47%
The AI correctly identifies the image as a dog with 94.32% confidence, while giving low scores to cat and car.
Computer Vision is like giving a brain to a blind camera.
Question: What is the difference between Object Detection and Object Segmentation? Answer: Object Detection identify objects and draws a bounding box around them, while Object Segmentation classifies every single pixel to provide the exact shape and boundaries of the object.
Question: How do Vision Transformers (ViTs) differ from traditional CNNs? Answer: CNNs process images using local filters to detect features, while ViTs split the image into patches and treat them like tokens, allowing the model to capture global dependencies across the entire image.
Question: What are multimodal models like CLIP, and how can they identify objects without being specifically trained on those objects? Answer: Multimodal models like CLIP are trained on both images and text descriptions together. They learn to understand the relationship between visual features and language, so they can recognize objects by their text descriptions even if they weren't explicitly trained on those specific categories. This is called "zero-shot" classification.