From the definition of a magnitude

$$m_1 - m_2 = -2.5\log_{10}\frac{F_1}{F_2}$$

Imagine breaking the vertical extent of the atmosphere up into infinitely many thin, parallel slabs of varying composition and character. As a ray of light enters the atmosphere and traverses some path through these layers, the ray interacts with each layer.

The Beer-Lambert Law gives us the relationship between the intensity of light rays entering a medium and the intensity of the resultant outgoing light rays. If the medium can be characterized with some extinction coefficient $\kappa$ in units of $\frac{\%}{m}$, then the change in intensity for every unit length of the medium traversed is $\Delta{I} = - \kappa I(x=0)\cdot \Delta{x}$, where $I(x=0)$ can be taken as the intensity of the light ray as it enters the medium.

In differential form, and allowing for $\kappa$ to depend on vertical location: $$\frac{dI}{dx} = -\kappa(x)I(x)$$

In the diagram above, you can see that any angle relative to zenith ($z$) actually forces the light ray to traverse a longer path through the atmosphere by a factor of $\sec{(z)}$. So the Beer-Lambert Law tells us that the differential amount of intensity lost to the extinctive character of the layers of the atmosphere is:

$$dI = -\kappa(x)I(x)\sec{(z)}dx$$

From here, we can note that $\frac{dI}{I} = d\ln{I}$. We will integrate over the path of the light ray: From the top of the atmosphere ($T$) to the bottom of the atmosphere, i.e., the ground ($B$).

Rearrange: $$d\ln{I} = -\kappa(x)\sec{(z)}dz$$ Integrate: $$\int_T^Bd\ln{I} = \int_T^B-\kappa(x)\sec{(z)}dx$$ We are left with: $$\ln{I(x=T)} - \ln{I(x=B)} = -\sec{(z)}\int_T^B\kappa(x)dx$$ Note that the $\int_T^B\kappa(x)dx$ cannot be known without careful (likely *in situ*) measurements. So let's just call this $K$. $$\ln{I(x=T)} - \ln{I(x=B)} = -\sec{(z)}K$$ By some property of logarithms: $$\ln{\frac{I(x=T)}{I(x=B)}} = -\sec{(z)}K$$

And so:

$$\frac{I(x=T)}{I(x=B)} = e^{-\sec{(z)}K}$$

From here, we must trust that $I \propto F$ in some way. See Radiative Quantities if that trust is hard to come by.

Given this proportionality, we can say that $I_1$ is the top-of-atmosphere intensity and that $I_2$ is the intensity that we register on our detector (reduced to the top-of-telescope) intensity.

$$m_T - m_B = -2.5\log_{10}\frac{I_T}{I_B}$$

Given the results from above, this means that: $$m_T - m_B = -2.5\log_{10}e^{-\sec{(z)}K}$$

We can clean this up a little algebraically:

$$m_T - m_B = -2.5(-\sec{(z)}K\cdot\log_{10}e)$$

$$m_T - m_B = 2.5\sec{(z)}K\cdot\log_{10}e$$

$$m_T = 2.5\sec{(z)}K\cdot\log_{10}e + m_B$$

So here is the work-up for the top-of-atmosphere magnitude based on $m_B$, $\sec{(z)}$, & $K$.

1. $m_B$ is the top-of-telescope instrument magnitude.
2. $\sec{(z)}$ is the secant of the angle between zenith and the observed source
3. $K$ is the atmosphere-integrated extinction

The first two are easy (ish) ! That last one, the atmosphere-integrated extinction is less easy. It must be derived either from measurements of the layer-by-layer extinction coefficient and numerically integrated OR we must find some way to measure the atmosphere-integrated extinction itself.

Take a look at the final result from above again: $$m_T = 2.5\sec{(z)}K\cdot\log_{10}e + m_B$$ Choose to rearrange this in the form of a standard $y=mx+b$ linear form with $m_B$ as the dependent variable: $$m_B = -K\Biggl(2.5\sec{(z)}\cdot\log_{10}e\Biggr) + m_T$$ With this, we can see that on a graph of $m_B$ vs $2.5\sec{(z)}\cdot\log_{10}e$, the magnitude of the slope will be $K$ and the y-intercept will be $m_T$.

Therefore, in order to measure $K$, we must observe a source for which $m_T$ is constant throughout the session and we must observe this source at a variety of zenith-angles, $z$. The top-of-telescope instrument magnitudes resulting from enough observations at differing zenith angles should trace a line. A fit to that line will result in a y-intercept of the top-of-atmosphere magnitude for the observed source and a slope that can serve as a reasonable estimate of the atmosphere-integrated extinction.

It's important to to note that the source we aim to understand as a part of the overall observing stands to be different from the source we observe to measure the extinction. Further, the source we observe to measure extinction should be non-variable in nature and appear in the same region of the sky as the source in question. This source we observe the measure extinction should be called the “extinction star”.