Computer Vision in Production: Deploying ML Models at Scale

Model Optimization for Production

Production computer vision models need to be fast. A model that achieves 95% accuracy in a notebook but takes 2 seconds per inference is useless for real-time applications. The optimization stack starts with model architecture selection — MobileNetV3 and EfficientNet-Lite are designed for edge deployment, while ResNet and Vision Transformers dominate when latency is less constrained.

Beyond architecture choice, three optimization techniques matter most: quantization (reducing weight precision from float32 to int8), pruning (removing low-impact neural connections), and distillation (training a smaller model to mimic a larger one). In our experience, INT8 quantization with TensorFlow Lite typically achieves 3–4× inference speedup with less than 1% accuracy loss on image classification tasks. This kind of optimization work sits at the center of our computer vision development practice.

Serving Infrastructure Choices

The serving infrastructure depends on your latency requirements and scale. For batch processing (document analysis, quality inspection), a simple Flask or FastAPI server behind a load balancer works. For real-time inference (video analytics, mobile apps), you need dedicated model serving infrastructure.

TensorFlow Serving, TorchServe, and Triton Inference Server are the three production-grade options. Triton stands out for multi-framework support — it serves TensorFlow, PyTorch, ONNX, and TensorRT models through a unified API with dynamic batching, model ensemble pipelines, and GPU memory management built in.

Preprocessing Pipelines

Preprocessing is where most production CV systems fail silently. The preprocessing applied during training must be exactly replicated during inference — same normalization values, same resize algorithm, same color space conversion. We wrap preprocessing in a serialized pipeline (ONNX preprocessing ops or custom TF GraphDef) that's versioned alongside the model. This eliminates the "works in testing, fails in production" preprocessing mismatch.

Model Versioning and A/B Testing

Every model in production should have a version registry. We use MLflow's model registry with stage transitions (Staging → Production → Archived). Each model version is immutable — you never update a model in place; you register a new version and promote it. Traffic splitting between model versions (canary deployments for ML) lets you validate accuracy improvements against real production data before full rollout.

The key insight: model A/B testing requires infrastructure that can route requests to different model versions based on user ID or traffic percentage, collect predictions from both versions, and compare business metrics (not just accuracy metrics) between them.

Detecting and Handling Data Drift

Data drift — when production data diverges from training data — is the silent killer of computer vision models. A model trained on daytime images degrades when deployed for nighttime surveillance. A defect detection model trained on summer production data fails in winter when lighting changes.

We monitor drift using statistical tests (Kolmogorov-Smirnov for feature distributions, population stability index for prediction distributions) on a sample of production inputs. When drift exceeds thresholds, we trigger retraining pipelines automatically. The retraining pipeline pulls the latest labeled data, validates it against quality checks, and produces a new model candidate.

Scaling Computer Vision Workloads

GPU inference scaling is fundamentally different from CPU web service scaling. GPU memory is finite and model loading takes seconds. Auto-scaling GPU inference servers requires careful configuration: scale based on queue depth (pending inference requests), not CPU utilization. We use KEDA (Kubernetes Event-Driven Autoscaling) with Prometheus metrics from Triton to scale GPU nodes based on inference queue length.

For edge deployments — cameras, drones, factory equipment — model distribution and OTA updates are the scaling challenge. We use a hub-and-spoke model: a central model registry pushes new model versions to edge devices during maintenance windows, with rollback capability if the new model's on-device validation fails. Our MLOps services team builds and manages this pipeline, and clients scaling their own ML teams often hire ML engineers through us to run it in-house.