Yeah, that seems absolutely true. Marx couldn't or wouldn't observe that there is always work involved in achieving profit. My guess is that he desperately wanted to prove that capitalism is inherently exploitive, so he worked his logic backward to come up with his labor theory of value.

The boss man gives the worker less than the worker gives the boss man, hence profit for the capitalist.