From the definition of a magnitude

$$m_1 - m_2 = -2.5\log_{10}\frac{F_1}{F_2}$$

Imagine breaking the vertical extent of the atmosphere up into infinitely many thin, parallel slabs of varying composition and character. As a ray of light enters the atmosphere and traverses some path through these layers, the ray interacts with each layer.

The Beer-Lambert Law gives us the relationship between the intensity of light rays entering a medium and the intensity of the resultant outgoing light rays. If the medium can be characterized with some extinction coefficient $\kappa$ in units of $\frac{\%}{m}$, then the change in intensity for every unit length of the medium traversed is $\Delta{I} = - \kappa I(x=0)\cdot \Delta{x}$, where $I(x=0)$ can be taken as the intensity of the light ray as it enters the medium.

In differential form, and allowing for $\kappa$ to depend on vertical location: $$\frac{dI}{dx} = -\kappa(x)I(x)$$

In the diagram above, you can see that any angle relative to zenith ($z$) actually forces the light ray to traverse a longer path through the atmosphere by a factor of $\sec{(z)}$. So the Beer-Lambert Law tells us that the differential amount of intensity lost to the extinctive character of the layers of the atmosphere is:

$$dI = -\kappa(x)I(x)\sec{(z)}dx$$

From here, we can note that $\frac{dI}{I} = d\ln{I}$. We will integrate over the path of the light ray: From the top of the atmosphere ($T$) to the bottom of the atmosphere, i.e., the ground ($B$).

Rearrange: $$d\ln{I} = -\kappa(x)\sec{(z)}dz$$ Integrate: $$\int_T^Bd\ln{I} = \int_T^B-\kappa(x)\sec{(z)}dx$$ We are left with: $$\ln{I(x=T)} - \ln{I(x=B)} = -\sec{(z)}\int_T^B\kappa(x)dx$$ Note that the $\int_T^B\kappa(x)dx$ cannot be known without careful (likely *in situ*) measurements. So let's just call this $K$. $$\ln{I(x=T)} - \ln{I(x=B)} = -\sec{(z)}K$$ By some property of logarithms: $$\ln{\frac{I(x=T)}{I(x=B)}} = -\sec{(z)}K$$

And so:

$$\frac{I(x=T)}{I(x=B)} = e^{-\sec{(z)}K}$$

From here, we must trust that $I \propto F$ in some way. See Radiative Quantities if that trust is hard to come by.

Given this proportionality, we can say that $I_1$ is the top-of-atmosphere intensity and that $I_2$ is the intensity that we register on our detector (reduced to the top-of-telescope) intensity.

$$m_T - m_B = -2.5\log_{10}\frac{I_T}{I_B}$$

Given the results from above, this means that: $$m_T - m_B = -2.5\log_{10}e^{-\sec{(z)}K}$$

We can clean this up a little algebraically:

$$m_T - m_B = -2.5(-\sec{(z)}K\cdot\log_{10}e)$$

$$m_T - m_B = 2.5\sec{(z)}K\cdot\log_{10}e$$

$$m_T = 2.5\sec{(z)}K\cdot\log_{10}e + m_B$$

So here is the work-up for the top-of-atmosphere magnitude based on $m_B$, $\sec{(z)}$, & $K$.

1. $m_B$ is the top-of-telescope instrument magnitude.
2. $\sec{(z)}$ is the secant of the angle between zenith and the observed source
3. $K$ is the atmosphere-integrated extinction

The first two are easy (ish) ! That last one, the atmosphere-integrated extinction is less easy. It must me derived either from measurements of the layer-by-layer extinction coefficient and numerically integrated OR we must find some way to measure the atmosphere-integrated extinction itself.

Take a look at the final result from above again: $$m_T = 2.5\sec{(z)}K\cdot\log_{10}e + m_B$$ Choose to rearrange this in the form of a standard $y=mx+b$ linear form with $m_B$ as the dependent variable: $$m_B = -K\Biggl(2.5\sec{(z)}\cdot\log_{10}e\Biggr) + m_T$$ With this, we can see that on a graph of $m_B$ vs $2.5\sec{(z)}\cdot\log_{10}e$, the magnitude of the slope will be $K$ and the y-intercept will be $m_T$.

Therefore, in order to measure $K$, we must observe a source for which $m_T$ is constant throughout the session and we must observe this source at a variety of zenith-angles, $z$. The top-of-telescope instrument magnitudes resulting from enough observations at differing zenith angles should trace a line. A fit to that line will result in a y-intercept of the top-of-atmosphere magnitude for the observed source and a slope that can serve as a reasonable estimate of the atmosphere-integrated extinction.

It's important to to note that the source we aim to understand as a part of the overall observing stands to be different from the source we observe to measure the extinction. Further, the source we observe to measure extinction should be non-variable in nature and appear in the same region of the sky as the source in question. This source we observe the measure extinction should be called the “extinction star”.

Sources

1. https://slittlefair.staff.shef.ac.uk/teaching/phy217/lectures/principles/l04/