Biological Inspiration

• Neurons: Basic processing units
• Synapses: Weighted connections
• Learning: Adapting connection strengths
• Parallel processing: Massive connectivity

Artificial Counterpart

• Perceptrons: Mathematical neurons
• Weights: Learnable parameters
• Training: Gradient-based optimization
• Layers: Organized processing units

Linear Models

• Limited to linear decision boundaries
• Cannot solve XOR problem
• Restricted representational power
• Simple but insufficient for complex data

Neural Networks

• Non-linear decision boundaries
• Universal approximation capability
• Hierarchical feature learning
• Scalable to complex problems

Step Function (Original)

Sigmoid Function (Modern)

Original Perceptron Rule

Gradient Descent (Modern)

Sigmoid Function

• Range: $(0, 1)$
• Smooth and differentiable
• Output interpretable as probability

Hyperbolic Tangent

• Range: $(-1, 1)$
• Zero-centered output
• Steeper gradients than sigmoid

ReLU (Rectified Linear Unit)

• Computationally efficient
• No vanishing gradient for $x > 0$
• Sparse activation
• Most popular choice

Leaky ReLU

• Avoids "dying ReLU" problem
• Small gradient for negative inputs
• Typically $\alpha = 0.01$

Common Issues

• Sigmoid/Tanh derivatives $\rightarrow 0$ for large inputs
• Deep networks suffer from this
• Solution: ReLU activations

• Neurons get stuck at zero output
• No gradient flows through
• Solution: Leaky ReLU, initialization

Matrix Dimensions

• Input: $\mathbf{a}^{(l-1)}$ has shape $(n_{l-1}, 1)$
• Weights: $\mathbf{W}^{(l)}$ has shape $(n_l, n_{l-1})$
• Output: $\mathbf{a}^{(l)}$ has shape $(n_l, 1)$

• Input batch: $\mathbf{A}^{(l-1)}$ has shape $(n_{l-1}, m)$
• Output batch: $\mathbf{A}^{(l)}$ has shape $(n_l, m)$
• where $m$ is the batch size

Parameter Count

• Network depth
• Layer width
• Batch size

Depth vs Width

• More layers, fewer neurons per layer
• Better feature hierarchies
• Can represent more complex functions
• Risk: vanishing gradients

• Fewer layers, more neurons per layer
• More parameters at each level
• Easier to train
• Risk: overfitting

Memory Considerations

• Store all activations $\mathbf{a}^{(l)}$
• Store all pre-activations $\mathbf{z}^{(l)}$
• Needed for backpropagation

• Larger batches: More memory, better GPU utilization
• Smaller batches: Less memory, more gradient noise

Given

Step 1: Hidden Layer

Step 2: Hidden Activations

Step 3: Output Layer

Gradient Computations

Output Layer

Computational Complexity

• Same order as forward pass
• Very efficient vs numerical gradients

• Must store all activations
• Memory scales with depth

Why Backpropagation Works

• Efficiency: Reuses computations via chain rule
• Automatic: No manual gradient derivation
• Exact: Computes exact gradients
• General: Works for any differentiable network

• Rumelhart, Hinton, Williams (1986)
• Made deep learning practical

Output Layer Error

Chain Rule Application

Error Propagation

Weight and Bias Gradients

L2 Regularization (Ridge)

• Shrinks weights towards zero
• Uniform penalty on all weights
• Smooth weight distributions
• Preferred for most applications

L1 Regularization (Lasso)

• Promotes sparsity (many weights → 0)
• Automatic feature selection
• Creates sparse networks
• Useful for interpretability

When to Use L2

• General-purpose regularization
• All features potentially relevant
• Want smooth weight shrinkage
• Most common choice

When to Use L1

• Feature selection needed
• Many irrelevant features
• Want sparse models
• Interpretability important

Dropout Benefits

• Prevents overfitting: Reduces complex co-adaptations
• Model averaging: Approximates ensemble of networks
• Robust features: Forces redundant representations
• Easy to implement: Simple modification to forward pass

Choosing Regularization

• L2 regularization ($\lambda = 0.01$)
• Dropout (rate = 0.5)
• Early stopping

• Increase regularization strength
• Add more dropout
• Reduce model complexity

Other Techniques

• Monitor validation loss
• Stop when it starts increasing
• Simple and effective

• Artificially increase training data
• Add noise, rotations, etc.
• Domain-specific techniques

Poor Initialization

Good Initialization

Learning Rate Selection

• Loss explodes or oscillates
• Network doesn't converge
• Weights become very large

• Training takes forever
• Gets stuck in local minima
• Poor final performance

Advanced Optimizers

Loss Monitoring

• Training loss: Should decrease monotonically
• Validation loss: Should decrease, then stabilize
• Gap: Indicates overfitting if too large

• Loss increases: Learning rate too high
• Loss plateaus early: Learning rate too low
• Validation loss increases: Overfitting

Gradient Monitoring

• Gradient norms: Should be reasonable ($10^{-6}$ to $10^{-1}$)
• Vanishing: Gradients → 0 in early layers
• Exploding: Gradients become very large

Activation Monitoring

• Activation statistics: Mean, std, sparsity
• Dead neurons: Always output zero
• Saturated neurons: Always in saturation region

• Reasonable variance (not too small/large)
• Some sparsity (for ReLU)
• No layers completely dead

Weight Monitoring

• Weight distributions: Should be reasonable
• Weight updates: $|\Delta W| / |W| \approx 10^{-3}$
• Layer-wise learning rates: May need adjustment

Problem: Vanishing Gradients

• Early layers don't learn
• Gradients approach zero

• Use ReLU activations
• Proper weight initialization
• Batch normalization

Problem: Overfitting

• Training accuracy >> validation accuracy
• Validation loss increases

• Add regularization (L2, dropout)
• Reduce model complexity
• More training data

Problem: Exploding Gradients

• Loss becomes NaN
• Weights blow up

• Gradient clipping
• Lower learning rate
• Better initialization

Problem: Slow Convergence

• Loss decreases slowly
• Gets stuck in plateaus

• Increase learning rate
• Use adaptive optimizers

Core Concepts

• Perceptron: Basic building block
• Multi-layer: Enable complex mappings
• Activation functions: Provide non-linearity
• Forward propagation: Compute predictions
• Backpropagation: Compute gradients efficiently
• Regularization: Prevent overfitting

Mathematical Foundation

• Matrix operations for efficiency
• Chain rule for gradient computation
• Optimization theory for training
• Probability theory for interpretation

Best Practices

• Architecture: Start simple, add complexity gradually
• Initialization: Xavier/He for proper gradient flow
• Optimization: Adam optimizer with proper learning rate
• Regularization: L2 + Dropout for generalization
• Monitoring: Track loss, gradients, activations
• Debugging: Systematic approach to problems

When to Use Neural Networks

• Large datasets available
• Complex non-linear patterns
• End-to-end learning desired
• Feature engineering is difficult

Computer Vision

• Image classification: ResNet, EfficientNet
• Object detection: YOLO, R-CNN
• Segmentation: U-Net, Mask R-CNN
• Face recognition: DeepFace, FaceNet
• Medical imaging: Cancer detection, radiology

Natural Language Processing

• Language models: GPT, BERT, T5
• Translation: Google Translate, DeepL
• Chatbots: ChatGPT, virtual assistants
• Text analysis: Sentiment, summarization

Other Domains

• Speech: Recognition, synthesis, processing
• Recommendation: Netflix, Amazon, Spotify
• Games: AlphaGo, OpenAI Five, StarCraft
• Robotics: Control, perception, planning
• Finance: Trading, fraud detection, risk
• Science: Drug discovery, climate modeling

Emerging Areas

• Generative AI: DALL-E, Midjourney, Stable Diffusion
• Multimodal: CLIP, GPT-4V
• Reinforcement Learning: Autonomous systems
• Scientific Computing: Physics, chemistry, biology

Specialized Architectures

• Convolutional Neural Networks (CNNs)
  • Spatial structure exploitation
  • Translation invariance
  • Computer vision applications
• Recurrent Neural Networks (RNNs)
  • Sequential data processing
  • Memory and temporal dynamics
  • LSTM, GRU variants
• Transformer Networks
  • Attention mechanisms
  • Parallel processing
  • Modern NLP backbone

Advanced Techniques

• Batch Normalization
  • Internal covariate shift
  • Training acceleration
• Residual Connections
  • Very deep networks
  • Gradient flow improvement
• Attention Mechanisms
  • Selective focus
  • Long-range dependencies
• Generative Models
  • VAEs, GANs, Diffusion
  • Creative AI applications