top of page

WoofLens

Animals Communicator

Context

01

Why Does My Dog Keep Barking Non-Stop?

Dogs communicate constantly through posture, movement, and expression, but much of this language goes unnoticed by humans. This project uses Snap augmented reality glasses to reveal those hidden signals in real time, translating canine emotions and body language into visual cues the wearer can understand. By making the invisible visible, the lens reimagines how humans learn to listen to dogs, enabling safer interactions, more intuitive training, and a deeper shared understanding.

How Big is the Market?

The opportunity sits at the intersection of the global pet technology, pet training, and animal behavior interpretation markets. Pet owners already spend billions annually on training services, behavior tools, and interpretive experiences—including premium offerings that claim to translate or explain pet emotions and intent. As pets are increasingly treated as family members, demand continues to grow for technologies and services that promise deeper understanding, safer interaction, and improved communication between humans and animals.
Screenshot 2025-12-27 at 9.30.25 PM.png
Screenshot 2025-12-27 at 8.00.27 PM.png
Screenshot 2025-12-27 at 8.01.04 PM.png

Problem Statement

02

What are we solving?

“Instead of predicting abstract emotions, the system learns to recognize observable visual signals. Things like posture, facial tension, movement patterns, and maps them into interpretable emotional cues that humans can understand.”
Who's this for?
This kind of system could support different audiences in different ways.

1. Dog Owners: for this project, I intentionally focused on pet owners, because emotional attachment is where communication breaks down most often.

2. Dog Trainers: For trainers, this could become a data-supported training tool — tracking stress signals, engagement levels, or progress over time instead of relying purely on observation.

3. Vets: For vets, it could act as a visual communication aid — especially for animals that can’t be examined easily or show stress only outside clinical settings.

Info

03

Project Topics:
  • AR glasses
  • Augmented Reality
  • Machine Learning
  • Snap Spectacles Lens
  • Canine Communication
  • Dog Training

​
Timeline:
2025 November - Now
Ongoing Project

 
Context:
  • Snap Spectacles Development
  • Penn Computer Vision
  • Computer Vision Application
  • SnapML Experiments 

Phase 1:

Model Training

04

The emotion recognition model was trained as a multi-class image classifier with four emotional states:

1. happy
2. angry
3. sad
4. neutral

The initial training dataset consisted of approximately 1,000 labeled images per class, sourced from Kaggle, publicly available dog emotion dataset. Using this dataset, the model achieved a validation accuracy of 97%, indicating strong performance under controlled conditions.

During real-world testing, however, performance degradation was observed for dogs with longer or fluffier coats, where facial features and body cues were less visually distinct. This highlighted a dataset bias toward short-haired breeds. To address this limitation, additional training data focusing on long-haired and fluffy dogs were introduced to improve robustness and generalization. The expanded dataset improved the model’s ability to interpret emotions across a wider range of breeds, coat types, and visual conditions, resulting in more reliable real-time inference for diverse dogs encountered in everyday environments.
Pre Training
download (2).png
Model Architecture
model architecture (1).png
download (6).png
Area Under Curve:
0.5 = guessing
0.7-0.8 = decent
0.9 = strong

 
download (7).png
The ROC curves show that the model separates emotional states very well across thresholds, with all classes achieving AUCs above 0.94. This tells me the visual features the model learned are discriminative and reliable.
Real-Time Testing

Phase 2:

05

Model Optimization

Failure → Iteration → Improvement
Observed Failure
Despite high validation accuracy, real-world testing revealed a key limitation:
the model consistently underperformed on long-haired and fluffy dogs, often misclassifying emotions due to occluded facial features and altered silhouette cues.
​
Diagnosis
This exposed a dataset bias—the original training data overrepresented short-haired breeds, causing the model to rely heavily on facial contours and eye visibility that were less reliable for fluffy dogs.
​
Iteration
To address this, an additional dataset of long-haired and fluffy dog images was curated and incorporated into the training pipeline. These images emphasized:
  • Obscured facial landmarks
  • Fur-dominant silhouettes
  • Subtler posture-based emotional cues
The model was retrained with this expanded dataset while maintaining class balance.
​
Improvement
After retraining, the model demonstrated noticeably improved generalization across dog breeds and coat types, with more consistent emotion predictions in real-world AR testing scenarios. This iteration reinforced the importance of data diversity over raw accuracy metrics, especially for embodied AI systems operating outside controlled environments.
Screenshot 2026-01-06 at 12.02.08 AM.png
Screenshot 2026-01-06 at 2.21.34 AM.png
Screenshot 2026-01-06 at 1.14.26 AM.png

Phase 3:

Visualizing Trained Detection Model

06

Challenge

Lens Studio Incompatible

During deployment, the initial dog emotion classification model proved incompatible with Lens Studio due to unsupported ONNX operations introduced during normalization and broadcasting layers. As a result, the model could not be imported or executed reliably within the AR runtime. To resolve this, the model was retrained with a simplified, Lens-Studio-compatible architecture and adjusted preprocessing pipeline, ensuring stable ONNX export and real-time inference on device while preserving high classification accuracy.
Screenshot 2025-11-20 at 1.30.50 PM.png

Phase 4:

Prototyping with Lens Studio

07

Interaction UI made with figma: 4 classifications
 
happy
happy.png
hangry
food.png
sad
sad.png
neutral
neutral.png
Lens Development in Lens Studio
1. Integrated mood-based response images into the scanner prefabs to visually reflect detected emotional states.
​
2. Designed and implemented structured model prompts to analyze canine facial expressions and body language, covering facial expression, body posture, and overall emotional state with concise, specific outputs.
​
3. Processed and filtered model outputs to categorize emotional responses into defined mood classes.
​
4. Configured scanner prefab textures to dynamically represent emotional states, including sad, happy, neutral, and angry (predominantly food-motivated behavior).
Screenshot 2026-01-06 at 9.18.39 PM.png
Screenshot 2026-01-06 at 9.19.48 PM.png
Real Time Testing with pictures in LS Preview
accuracy and responses looked reliable so I went ahead and tested on real dogs on Spectacles

Phase 5:

Real Time Testing on Spectacles

08

In-development for more classifications and visualizations.....Stay Tuned

bottom of page