Blind at the Bottom

BDYWRK is an iOS app that counts squats, sit-ups, pull-ups, and push-ups using sensors in your phone. Three of those are tractable because the exercise moves the phone itself, so an accelerometer or a gyroscope gives you something to work with.

Push-ups can be counted clumsily from an accelerometer too, but vision is a more robust way to track them. The phone goes on the floor, screen up, and you do push-ups over it.

Since the front camera points straight up at you, the first implementation used the obvious signal: as you lower yourself, your body covers the lens and the image goes dark, and when you push back up the light returns, so a bright to dark to bright cycle is one rep. It needed no pose estimation, no machine learning, and no model to ship, because the whole input was the average brightness of the frame.

That is what I built first, and this is the story of why it was not enough and of the rebuilds it took to fix it.

1.0: counting the dips

The first detector was a state machine over one brightness value per frame that is smoothed out with an exponential moving average.

The smoothed signal is compared against a slowly drifting baseline of what "uncovered" looks like. Falling far enough below the baseline means you are on the way down, and climbing back near enough means you have come up, which is a rep. The two cutoffs are deliberately different numbers, because a signal hovering at a single threshold would cross back and forth on noise alone and turn one slow push-up into a dozen. That gap between them is hysteresis, the same trick a thermostat uses so it does not cycle the AC every few seconds.

Two timing constraints finish the job: a dip has to last a minimum duration, and consecutive reps have to be separated by a minimum gap. Both exist to rule out anything faster than a controlled push-up.

That is about forty lines of code, it runs in microseconds, and on a clean clip it genuinely works:

Even on a signal that clean, the thresholds still had to be tuned, so I tuned them hard with a grid search over 20,580 parameter combinations. On the clips it was tuned against, all of them full of push-ups, the best configuration scored a perfect F1 of 1.000.

Then I pointed it at a clip where nothing happens.

Why it was (obviously) not enough

No amount of tuning fixes this, because it is not a tuning problem. The detector's entire model of the world is "the frame got darker, then brighter," which fits a push-up, but also a passing shadow, or the auto-exposure hunting that is actually happening above. Across a limited labeled set it invents phantom reps in still rooms and around people moving without exercising. Counting the dips was the easy half, but deciding whether an oscillation is push-ups at all is the hard one, and one brightness channel could not make that call.

2.0: give it eyes

An obvious fix is to provide a signal to disambiguate the cases brightness could not: Apple's Vision framework hands you the position and confidence of every major body part, thirty times a second, for free.

The design was narrower than "find the push-up." Since there is nothing to see at the bottom of a push-up, pose never had to see the rep itself, only confirm the top. When you push back up, your body clears the lens and a shoulder should reappear, so requiring a keypoint at the up-transition checks that a real body returned, rather than just a dip in brightness.

The logic was sound, and the detector got worse anyway even after a similar tuning cycle: both precision and recall fell, while phantom reps on still-room clips went from 1.2 to 2.6.

Understanding why is the part of this project I would keep if I had to throw away the rest.

1.2 percent: how often pose saw a body during a rep

The table below shows the fraction of frames on one labeled clip where any keypoint cleared 0.05 confidence.

The table shows pose failing in both directions at once. Inside a rep, pose is barely there: at the bottom of a push-up your chest is practically on top of the lens, and there is no body left to recognize. Between reps it fires no more often than it does for someone walking around with a water bottle, so my check was really answering "is a person in frame" rather than "did a rep just finish."

So pose did not sharpen the detector on its own, and it mostly complicated the rules instead. Should it gate brightness, veto it, or vote alongside it, and at what confidence floor? Adding another signal did not add a rule so much as complexity and ambiguity.

Here are both signals on one clip, a set of seven labeled reps over twenty-two seconds. It is a different clip from the table above, and the same pattern holds: about 2.5% of frames inside reps have a keypoint, against 29% between them.

No better pose model fixes that, since nothing recognizes a body part closer than the lens's minimum focal distance. The failure is geometric, not algorithmic.

3.0: stop writing the rules

Both previous versions failed the same way: every rule I hand-wrote was a rule about one instant, such as "is it dark now" or "is a shoulder visible now." But a push-up is not an instant: it is a set of movements about a second and a half long. So instead of coding conditional patterns I allowed a deep learning model to learn from the labeled data which signals separate a rep from everything else.

I started with a simple baseline: a random forest, which trains in seconds on minimal data. But per-frame decisions are fundamentally flawed: a single dark frame looks identical whether it belongs to a push-up or a passing shadow. Fixing this meant hand-crafting temporal features (perhaps for an XGBoost model), which drops you right back into guessing which time windows matter and maintaining the pipeline to encode them.

That leaves the models that read a stretch of time natively. A small convolutional network or a transformer re-reads a fresh window of frames on every new frame, while a recurrent model carries one small running state and updates it once. Either family of models would likely have worked, but the recurrent one was simply the better fit for something that has to run live on the phone (as a CoreML streaming model), beside the camera and the pose estimator, at a few hundred kilobytes. A transformer also wants more training examples than I was confident I could supply, and examples were the thing I had least of.

So I built an LSTM that reads sixteen numbers per frame (brightness, plus the confidence and position of a handful of body landmarks) and emits a per-frame probability that a rep is in progress, which a threshold with hysteresis turns into counts. The same state machine sits at the end as before, the difference being that what feeds it has seen the last couple of seconds rather than the last frame. Handing it the pose landmarks after two sections spent showing how blind they are may look like a contradiction, and it is a fair one to raise: whether the model needs all sixteen numbers, or brightness plus two landmarks would do, is an ablation I have not run yet, and it is on the list.

And it finds things I never told it to look for. Nowhere did I say that a rep is a dark stretch of a certain length bracketed by light, or that the tracked body parts vanish on the way down and reappear at the top. It found that in the data, across seconds rather than frames, which is the view my hand-written rules never had. The price is that it wants examples rather than arguments.

Three generations, one clip set

The three detectors had never faced the same test, since each had been evaluated on whichever clips it happened to support. So I rebuilt the comparison as a fair one: all three versions, the same 89 clips, the same hand-labeled reps, and the same rule for what counts as a match, meaning a detected rep has to land within half a second of a labeled rep to be credited.

F1 is a score that rewards finding the reps that happened and punishes inventing reps that did not, and 1.000 is perfect. Adding pose did not merely fail to help but actively made the score worse, and only the model that can see time gets anywhere near respectable.

Researchers care about F1, but users only care that the final rep count on their screen is right. Across the 67 clips that actually contain push-ups, averaging 10.6 reps each, brightness and the pose version both finish a set off by 7.4 reps. The LSTM, however, is off by an average of 2.0 reps, with a median miss of just 1.

Beyond counting real push-ups, the LSTM also handles phantom reps far better, though it does not eliminate them. The camera is easy to fool, whether by someone resting between sets or by a shadow cast from a person walking past. Watching a still room, brightness invents 1.2 reps per clip and the pose version 2.6, while the LSTM invents two reps total across the same clips. When someone is moving around but not exercising, the counts run 1.5, then 2.5, then two total across all twelve. That is a 6x reduction in phantom reps in a still room, and 9x when something else is moving. That gap matters more than the miss rate does, because a user forgives an occasional missed rep far more readily than reps their app added on its own.

That accuracy comes with a slight trade-off in speed. Because the LSTM needs context before it can decide, it takes a median 1.30 seconds to confirm a rep, against 1.04 for the bare brightness detector. Reducing that lag is on the list, but accuracy is the priority, and a count that lands a beat late is always better than one that lands wrong.

What four grid searches taught me about my dataset

The model exists to reject nuisance motion, so every validation fold contains a clip with real push-ups plus clips of nothing and clips without exercise. Then I ran the same search four times, at four sizes of the growing dataset: sixty-four configurations each round, with the top five retrained across six holdout splits so a winner could not be a lucky draw. The clip counts below are the dataset as it stood at each search, not the 89-clip set from the comparison above.

Across all four searches the winner kept moving in one direction (barring the rise between the 49 and 70 clip searches, where added clips were more diverse and difficult). Otherwise, small datasets rewarded simpler models, and as the dataset grew to 86 clips, it could finally support a wider, more complex model, reined in by regularization, without overfitting. Ultimately, these tests measured data volume rather than architecture.

The spread bars reveal a similar volatility: simply changing which clips land in the validation split causes a 2x performance gap, meaning any single run can flatten the results. The honest conclusion is that labeling more data is the only lever that matters, and the 89 clips and 709 hand-labeled reps I already have are an encouraging start.

Where it stands, and what comes next

This is the detector in the App Store build today: when the count on screen ticks up, this model moved it. The most valuable part of this baseline is exposing its precise limits: it fails entirely in the dark, and testing so far shows it generalizing across indoor lighting but not yet across different bodies. These known flaws yield a compounding, 26-item roadmap prioritizing data diversity, input cleanup, and incorporating other phone sensors for when vision drops out.

The post I ultimately want to write is about a model trained on a thousand clips. It requires the unglamorous step of manually labeling a thousand sessions, but putting in that foundation now guarantees the detector is as bad today as it will ever be. Once this baseline is established, it will bootstrap my autolabeler to rapidly scale and drive all future iterations.

The transferable part

The specific lesson here is that the model acts as a nuisance discriminator, not a counter. Forty lines of signal processing can count dips, while deciding whether those movements are actually push-ups is what requires a labeled dataset and a time-aware model. More broadly, architecture search cannot compensate for a lack of signal. Tuning simply ranks configurations against each other, which means even flawed features will produce a "winner" that creates the illusion of progress. The only way to catch that is to scrutinize your own inputs and look closely at the actual data, which here meant the 1.2% table above and then seeing for myself why pose went blind.

The workflow was simple: build fast, then refine. I pushed the most basic solution to its limit, forcing any added complexity to earn its keep. The meaningless perfect score early on served as a warning that evaluation metrics have to evolve alongside the detector they are grading. Moving from hand-written rules to a trained model ultimately paid off faster than I expected. The current performance is not the ceiling either, because every batch of new clips has moved it, and there is no reason for that to stop.

---

A privacy note: BDYWRK never records video or photos. Each camera frame is reduced on the spot to the sixteen numbers the model reads, then discarded, and the labeled clips behind this post came from a developer-only capture mode running on my own phone. See the Privacy page for more information.

BDYWRK counts push-ups, pull-ups, sit-ups, squats, and more, using your iPhone's sensors, with everything running on-device. Get BDYWRK now, exclusively on the App Store.