Testing Methodology
Our testing procedures are designed to provide consistent, reproducible results that reflect real-world performance. This document details our approach for each product category.
General Testing Principles
Product Acquisition
All products are purchased at retail price from standard consumer channels (Amazon, manufacturer websites, retail stores). We do not accept free review units or loaner devices. This ensures we receive the same product quality as consumers.
Testing Duration
Minimum testing period is 30 days for most products. Long-term reliability testing extends to 90 days for products where durability is a key factor. This allows us to identify issues that may not appear in short-term testing.
Environmental Control
Standardized tests are conducted in controlled environments (68-72°F, 40-60% humidity) to ensure consistency. Real-world testing occurs in various conditions to validate performance across expected use cases.
Smart Speakers
Audio Quality Assessment
We use calibrated measurement microphones and analysis software to measure:
- Frequency response (20Hz-20kHz) at 1 meter in anechoic conditions
- Maximum volume before audible distortion (THD > 1%)
- Bass extension and low-frequency accuracy
- Stereo imaging for multi-driver systems
Voice Assistant Testing
Standardized command recognition testing includes:
- 100 common commands spoken from various distances (1m, 3m, 5m)
- Recognition accuracy with background noise (music, conversation)
- Response latency measurement
- Multi-room command handling (where applicable)
Smart Home Integration
We test compatibility with major smart home platforms and measure command execution reliability across 500+ commands over the testing period.
Robot Vacuums
Cleaning Performance
Standardized debris pickup tests on multiple surfaces:
- Hardwood floors: Rice, cereal, pet hair, fine dust
- Low-pile carpet: Same debris types plus tracked-in dirt
- High-pile carpet: Embedded debris extraction capability
- Edge cleaning: Performance along walls and corners
Each test uses precisely measured quantities of debris. Pickup percentage is calculated by weighing remaining debris after three cleaning passes.
Navigation & Mapping
We evaluate mapping accuracy and navigation efficiency:
- Coverage rate in standardized room layout (% of floor reached)
- Obstacle avoidance success rate (50 obstacle tests per product)
- Return-to-dock success rate after 100 cleaning cycles
- Multi-room transition reliability
Battery & Runtime
Battery life measured under standardized cleaning conditions. We record runtime until automatic dock return across 20 complete discharge cycles and report the median value.
Wireless Earbuds
Audio Quality
Measurements conducted using artificial ear canal and calibrated microphones:
- Frequency response: Deviation from Harman target curve
- Total harmonic distortion: At 90dB SPL across frequency range
- Channel matching: Left/right driver consistency
- Codec performance: Comparison of AAC, aptX, LDAC (when supported)
Active Noise Cancellation
ANC effectiveness measured using calibrated noise sources:
- Attenuation across frequency spectrum (50Hz-8kHz)
- Real-world effectiveness (airplane cabin, office, street traffic)
- Transparency mode accuracy and natural sound quality
Battery Life
Continuous playback at 75dB SPL with AAC codec. Both ANC-on and ANC-off modes tested separately. Case charging cycles validated across 20 complete charge/discharge cycles.
Smart Thermostats
Temperature Accuracy
Sensor accuracy verified against NIST-traceable reference thermometers:
- Temperature reading accuracy (±0.5°F target)
- Response time to temperature changes
- Sensor consistency across room sensors (where included)
Energy Savings Validation
90-day controlled test in residential environment with HVAC runtime monitoring. Comparison against baseline thermostat with identical schedule. Energy savings calculations include heating and cooling cycles.
Learning & Scheduling
For learning thermostats, we evaluate schedule optimization over 30 days. Assessment includes comfort maintenance while reducing unnecessary runtime.
Fitness Trackers
Heart Rate Accuracy
Validation against medical-grade chest strap monitors:
- Resting heart rate: 50 five-minute measurements
- Exercise heart rate: Treadmill test with gradual intensity increase
- Recovery tracking: Post-exercise heart rate decline accuracy
- Irregular rhythm detection: Comparison with medical ECG
GPS Accuracy
Outdoor activity tracking validated against survey-grade GPS receivers. Testing includes open-sky conditions and urban canyon scenarios. Distance accuracy reported as percentage error over 5km routes.
Battery Life
Real-world usage simulation with standardized daily activity profile. Includes typical notification volume, display-on time, and sensor usage. Battery life validated across three complete charge cycles.
Scoring Methodology
We do not assign overall scores or star ratings. Instead, we report measured performance metrics and provide context for their practical significance.
Where subjective assessment is necessary (comfort, ease of use), we clearly identify it as subjective and explain our reasoning. Multiple team members evaluate subjective factors to reduce individual bias.
Our recommendations consider measured performance, reliability, value, and specific use cases. We avoid declaring a single "best" product, recognizing that optimal choice depends on individual needs and priorities.
Methodology Updates
We continuously refine our testing procedures based on emerging technology, user feedback, and methodological advances. When we make significant changes to our testing approach, we note this in affected reviews and, where appropriate, retest products under the new methodology.
Last updated: 16 February 2026
Questions about our methodology? Contact us
Learn more about our mission