top of page
WoofLens
Animals Communicator
Context
01
Why Does My Dog Keep Barking Non-Stop?
Dogs communicate constantly through posture, movement, and expression, but much of this language goes unnoticed by humans. This project uses Snap augmented reality glasses to reveal those hidden signals in real time, translating canine emotions and body language into visual cues the wearer can understand. By making the invisible visible, the lens reimagines how humans learn to listen to dogs, enabling safer interactions, more intuitive training, and a deeper shared understanding.
How Big is the Market?
The opportunity sits at the intersection of the global pet technology, pet training, and animal behavior interpretation markets. Pet owners already spend billions annually on training services, behavior tools, and interpretive experiences—including premium offerings that claim to translate or explain pet emotions and intent. As pets are increasingly treated as family members, demand continues to grow for technologies and services that promise deeper understanding, safer interaction, and improved communication between humans and animals.



Problem Statement
02
What are we solving?
“Instead of predicting abstract emotions, the system learns to recognize observable visual signals. Things like posture, facial tension, movement patterns, and maps them into interpretable emotional cues that humans can understand.”
Who's this for?
This kind of system could support different audiences in different ways.
1. Dog Owners: for this project, I intentionally focused on pet owners, because emotional attachment is where communication breaks down most often.
2. Dog Trainers: For trainers, this could become a data-supported training tool — tracking stress signals, engagement levels, or progress over time instead of relying purely on observation.
3. Vets: For vets, it could act as a visual communication aid — especially for animals that can’t be examined easily or show stress only outside clinical settings.
Info
03
Project Topics:
-
AR glasses
-
Augmented Reality
-
Machine Learning
-
Snap Spectacles Lens
-
Canine Communication
-
Dog Training
Timeline:
2025 November - Now
Ongoing Project
Context:
-
Snap Spectacles Development
-
Penn Computer Vision
-
Computer Vision Application
-
SnapML Experiments
Phase 1:
Model Training
04
The emotion recognition model was trained as a multi-class image classifier with four emotional states:
1. happy
2. angry
3. sad
4. neutral
The initial training dataset consisted of approximately 1,000 labeled images per class, sourced from Kaggle, publicly available dog emotion dataset. Using this dataset, the model achieved a validation accuracy of 97%, indicating strong performance under controlled conditions.
During real-world testing, however, performance degradation was observed for dogs with longer or fluffier coats, where facial features and body cues were less visually distinct. This highlighted a dataset bias toward short-haired breeds. To address this limitation, additional training data focusing on long-haired and fluffy dogs were introduced to improve robustness and generalization. The expanded dataset improved the model’s ability to interpret emotions across a wider range of breeds, coat types, and visual conditions, resulting in more reliable real-time inference for diverse dogs encountered in everyday environments.
Pre Training
.png)
Model Architecture
.png)
.png)
Area Under Curve:
0.5 = guessing
0.7-0.8 = decent
0.9 = strong
.png)
The ROC curves show that the model separates emotional states very well across thresholds, with all classes achieving AUCs above 0.94. This tells me the visual features the model learned are discriminative and reliable.
Real-Time Testing


Phase 2:
05
Model Optimization
Failure → Iteration → Improvement
Observed Failure
Despite high validation accuracy, real-world testing revealed a key limitation:
the model consistently underperformed on long-haired and fluffy dogs, often misclassifying emotions due to occluded facial features and altered silhouette cues.
Diagnosis
This exposed a dataset bias—the original training data overrepresented short-haired breeds, causing the model to rely heavily on facial contours and eye visibility that were less reliable for fluffy dogs.
Iteration
To address this, an additional dataset of long-haired and fluffy dog images was curated and incorporated into the training pipeline. These images emphasized:
-
Obscured facial landmarks
-
Fur-dominant silhouettes
-
Subtler posture-based emotional cues
The model was retrained with this expanded dataset while maintaining class balance.
Improvement
After retraining, the model demonstrated noticeably improved generalization across dog breeds and coat types, with more consistent emotion predictions in real-world AR testing scenarios. This iteration reinforced the importance of data diversity over raw accuracy metrics, especially for embodied AI systems operating outside controlled environments.



Phase 3:
Visualizing Trained Detection Model
06
Challenge
Lens Studio Incompatible
During deployment, the initial dog emotion classification model proved incompatible with Lens Studio due to unsupported ONNX operations introduced during normalization and broadcasting layers. As a result, the model could not be imported or executed reliably within the AR runtime. To resolve this, the model was retrained with a simplified, Lens-Studio-compatible architecture and adjusted preprocessing pipeline, ensuring stable ONNX export and real-time inference on device while preserving high classification accuracy.

Phase 4:
Prototyping with Lens Studio
07
Interaction UI made with figma: 4 classifications
happy

hangry

sad

neutral

Lens Development in Lens Studio
1. Integrated mood-based response images into the scanner prefabs to visually reflect detected emotional states.
2. Designed and implemented structured model prompts to analyze canine facial expressions and body language, covering facial expression, body posture, and overall emotional state with concise, specific outputs.
3. Processed and filtered model outputs to categorize emotional responses into defined mood classes.
4. Configured scanner prefab textures to dynamically represent emotional states, including sad, happy, neutral, and angry (predominantly food-motivated behavior).


Real Time Testing with pictures in LS Preview

accuracy and responses looked reliable so I went ahead and tested on real dogs on Spectacles
Phase 5:
Real Time Testing on Spectacles
08


In-development for more classifications and visualizations.....Stay Tuned
bottom of page