PCHMR: Empowering and Benchmarking Human Mesh Recovery in Privacy-Constrained Real-World Settings

Human pose and body shape are already supported by major personal-device platforms and applications. Meta Quest tracks body motion to animate VR avatars and provides pose-comparison tools for fitness exercises; Apple's Vision framework estimates 3D body joints for exercise tracking, sports analysis, and character animation; Google ML Kit supports mobile fitness-pose recognition and repetition counting; and Snapchat's Clothing Try-On uses a body mesh that follows a person's movement and shape. These systems can observe identifiable images, body geometry, and motion inside homes and other private spaces. Personalizing an HMR model may improve its fit to a particular user, camera, and environment, but collecting those images on a central server creates avoidable exposure. PCHMR studies how HMR models can be trained and adapted while the images remain on the user's device.

Training Setup

PCHMR uses Flower to simulate 10, 100, and 1000 clients. Each global round samples 10 clients; all clients participate in the 10-client setting. Each sampled client trains for one local epoch, a single pass over its own data shard, before aggregation. The corresponding IID runs use 200, 2000, and 6000 global rounds. A local epoch, an optimizer step, and a global round describe different amounts of work; the training-budget analysis below makes those differences explicit.

Privacy setting comparison. PCHMR treats centralized HMR as a reference and evaluates federated training, in which raw images remain on each client, plus a secure-aggregation variant.

Federated training prevents the server from receiving raw images, but the updates and the released model may still reveal information. In the secure-aggregation experiment, the server follows the protocol but may try to inspect the updates; it can see only their aggregate, not each client's update. The study does not claim differential privacy and does not evaluate membership inference, gradient inversion, or model inversion attacks.

Training Budget and Evaluation

One epoch-equivalent is 476,062 sample-visits, the size of the full training set. The published centralized baselines use 60 epochs; the IID 10/10 and 100/10 runs receive 200 epoch-equivalents, while 1000/10 receives 60. The added 200-epoch centralized run matches sample-visits, but its effective batch of 512 produces far fewer optimizer steps than the federated local batch of 64. Equal data exposure therefore does not imply equal optimization work.

Training budgets and measured simulation cost. Selected settings from the paper's Training Budget appendix. Optimizer steps are summed over clients. The two timing columns refer to different backbones.

LDA work counts are averaged over partition and client-sampling seeds. Timings are medians over completed runs on a shared cluster with heterogeneous nodes; they describe simulation cost rather than a controlled hardware comparison or real deployment latency. The paper also projects about 500 hours for centralized training at batch 64, but that configuration was not run to completion and is not included among these measurements.

On narrow screens, scroll each plot horizontally to view all panels.

The centralized curves reach their best validation error around epochs 26–27, then flatten or worsen. This matters independently of equalizing data passes: extending the centralized schedule does not necessarily improve the reference model. The federated curves also show stronger evaluation-to-evaluation variation, especially with skewed client data.

Both regimes use the same Human3.6M protocol-2 validation split of 27,558 images, evaluated with batch size 256. Reported errors select the best checkpoint on that split; there is no separate held-out test set in this comparison. Centralized runs evaluate each epoch. Federated runs evaluate every round for 10/10 and every ten rounds for 100/10 and 1000/10, giving 200, 200, and 600 validation evaluations respectively.

DePoser Local Annotation

DePoser uses Sapiens 2D keypoints and joint-depth cues to guide SMPLify-X-based body fitting. The paper describes metric-depth alignment with a lightweight linear regressor and a scale-invariant logarithmic (SILog) depth objective. The reported experiment evaluates the combined annotation pipeline rather than isolating the contribution of each component.

L_SILog = √[(1/N) Σ_i d_i² − λ_si((1/N) Σ_i d_i)²],   d_i = log z_i − log ẑ_i.

Here z denotes pseudo-ground-truth joint depth and ẑ predicted joint depth. The paper uses λ_si = 0.5; subtracting the squared mean residual reduces the influence of a shared depth-scale difference.

Effect of the Number of Clients

These are the client-scale accuracy results reported in the paper. The 10/10 and 100/10 settings use more sample-visits than the published centralized baselines, while 1000/10 matches their data exposure but performs more optimizer steps. Read the numerical differences together with the budget analysis above; they do not isolate the effect of federated aggregation.

Client-scale evaluation. Human3.6M protocol-2 validation error in millimeters. Lower is better.

The 1000/10 setting has higher error than the published centralized baseline for both models. The 10/10 setting gives numerically lower errors, but receives 200 epoch-equivalents against the baseline's 60. The paper reports these accuracy values as point estimates; the learning curves and budget analysis provide additional context for their interpretation.

Uneven Client Data

The Dirichlet partitions use source-dataset labels to control how examples are distributed across clients; a smaller α produces stronger skew. Across α = 0.1 to 1000, the runs receive 199.8–200.0 epoch-equivalents and approximately 1.497 million optimizer steps. The low error at α = 0.1 therefore does not come from a larger training budget than the IID 100/10 run. The paper discusses aggregation-induced regularization as a possible explanation, without establishing it as the mechanism.

Skew changes who performs the work even when total work stays nearly constant. From α = 1000 to 0.1, the summed per-round workload of the slowest sampled client grows from 154,767 to 707,920 optimizer steps, about 4.6 times. The reported simulation time rises from 59.2 to 87.4 hours for FastMETRO-S and from 57.4 to 77.7 hours for TORE-S. This is the straggler cost of uneven client data.

Natural Partition on 3DPW

Natural scene partitions approximate real deployment, where each user's data is shaped by identity, clothing, environment, camera pose, and local data volume.

3DPW fine-tuning. IID fine-tuning uses 100 clients with 10 sampled per round; the natural partition assigns each scene to a client and uses 20 clients with 5 sampled per round. All errors are in millimeters; lower is better.

DePoser creates mesh labels from each user's images on that user's device, and those generated labels are used for local fine-tuning. For the in-the-wild examples below, the same labels are also used as the evaluation reference. The reported reductions therefore show that the fine-tuned model agrees more closely with DePoser; they do not establish higher 3D accuracy against independent measurements.

Personalization Results

Agreement with DePoser-generated references. All errors are in millimeters; lower values mean that the fine-tuned model is closer to the labels generated by DePoser. They do not measure accuracy against independent 3D ground truth.

DePoser and SMPLify-X. Errors in millimeters on a randomly sampled 300-image subset of 3DPW with independent 3D ground truth. This compares complete annotators. Lower is better.

The combined DePoser pipeline reduces MPVPE by 36.06 mm, MPJPE by 29.88 mm, and PA-MPJPE by 16.03 mm in this comparison. The experiment does not attribute those improvements separately to the keypoint detector, depth objective, or depth alignment.