What does the detector measure?

tl;dr: Photons. CCDs and CMOSs are photon counters.

The count value associated with a pixel in the raw light frame is a value that is proportional to the amount of charge that was built-up over integration time in that pixel. These are digital units derived from the on-chip analog-to-digital conversion process that is thought to be (to a first-order approximation) linear with respect to the voltage generated by electrons liberated from the material that composes the pixels of the detector.

So I've passed the question from “counts” to “voltage generated by liberated electrons”. Grant me that the piling up of electrons over the integration-time generates a voltage. After that, what liberates the electrons? Answer: Energy. This energy may come in one of a few forms. (1) From a voltage applied to a pixel, (2) from the temperature of the electrons, or (3) from the radiative energy associated with incident photons. For a more delicate discussion of this, refer to ideas in semiconductor physics.

The voltage applied to the pixel and any thermal agitations clearly arise from the detector itself. These contributions to the counts must be calibrated away (see Data Reduction I: Remove Detector Effects). After that, the counts speak to only the liberated electrons due to interactions with photons incident on the detector.

The energy associated with the liberation of these electrons is the radiative energy we seek to measure. This energy interacts with the semiconductor substrate and liberates photoelectrons at a rate assumed to be linear with respect to the integration-time.

$$E_{\gamma}~d\lambda = \frac{hc}{\lambda}d\lambda$$

Energy carries the normal units Joules ($[E]=~ J$)¹⁾, but be careful! These counts are proportional to the received radiant energy (in the form of photons for this discussion) incident on a single pixel. This single pixel receives photons from a specific angular area of the sky, the pixel has an area over which it received photons, an orientation of that area, a time during which it was 'allowed' to receive photons, and a sensitivity to photons over a specific range of wavelengths.

So these counts are proportional not to the total energy of a target observed by the telescope, but to the differential amount of that energy called (in the broadest sense) the spectral radiance.

$$I(\lambda, \theta, \phi) = \frac{d^6 E}{d\lambda\cdot dt\cdot d^2A \cos{\theta}\cdot d^2\Omega(\theta,\phi)}$$

The spectral radiance has units $[I(\lambda, \theta, \phi)]=J\cdot (m\cdot t\cdot m^2\cdot sr^2)^{-1}$

Said slightly differently, the spectral radiance is the amount of energy received from a specific angular area of the sky ($d^2\Omega(\theta,\phi)$), over an area ($d^2A$) with some orientation ($\cos{\theta}$), during time during which it was 'allowed' to receive energy ($dt$), and a sensitivity to photons over a specific range of wavelengths ($d\lambda$).

Each pixel effectively integrates this spectral radiance with respect to wavelength, time, perpendicular area, and solid angle. The resulting measure of radiant energy is what is responsible for liberating photoelectrons in the CMOS semiconductor substrate which are eventually readout as counts.

Since each of the pixels are assumed to be sensitive to photons of the same range of wavelengths, were exposed to photons for the same amount of time, have the same orientation and area, and look at congruent solid angles of the sky, we can appropriately sum-up the bits of radiant energy to think about the total radiant energy that fell on a pixel during our integration time.

Since we do know the range of wavelengths, the integration time, and pixel size, we can define the spectral flux:

$$F(\lambda) = \frac{d^4 E}{d\lambda\cdot dt\cdot d^2A_{\perp}}$$

The radiative, spectral flux has units $[F(\lambda)]=J\cdot (m\cdot t\cdot m^2)^{-1}$.