Differences

This shows you the differences between two versions of the page.

--- wiki:observational_astronomy:data_reduction [2023/11/09 12:55] – Roy Prouty
+++ wiki:observational_astronomy:data_reduction [2023/11/09 13:50] (current) – removed Roy Prouty
@@ Line 1: / Line 1: @@
-====== Data Reduction To Energy Incident at Telescope ======
-The term reduction refers to the removal/reduction of extra counts collected in light frames. An appropriately run data reduction pipeline accounts for the effects of the optical system, detector, and atmosphere. This page outlines the theory that can be applied to account for the effects of the optical system and detector.
-This page outlines the best practices for our system. It's currently a work in progress.
-=== Unwanted Instrument Signals ===
-  * [[wiki:observational_astronomy:calibration_frames:thermal_signal|Thermal Signal]]
-  * [[wiki:observational_astronomy:calibration_frames:bias_signal|Bias Signal]]
-  * [[wiki:observational_astronomy:calibration_frames:uniformity|Uniformity (flats)]]
-=== Theory & Background ===
-We can represent each $i$ and $j$-th pixels in a light frame as the sum of the contributions from the following signals:
-$$L_{ij}(\lambda) = \Bigl[\epsilon_{ij}(\lambda)I_{ij}(\lambda) + T_{ij}(\lambda)\Bigr]t + B_{ij}(\lambda)$$
-== Light ==
-$L_{ij}(\lambda)$ is measured in counts and is the signal shown in SharpCap and saved in the data portion of the FITS file. This is the raw data in any light frame.
-While not super important for this data reduction example, it's otherwise important to note that these counts are limited in their numerical precision and therefore in their ability to precisely represent continuous physical values, i.e., $L_{ij}(\lambda)\in \mathbb{Z}<2^{\tiny{BITPIX}}-1$.\\
-== Uniformity ==
-$\epsilon_{ij}(\lambda)$ is the pixel-by-pixel uniformity or deviation from uniformity. This is measured as a fraction with each pixel carrying a (to computer precision) continuous value in the range $\epsilon_{ij}(\lambda)\in \mathbb{R}_{[0,1]}$. In an ideal system, $\epsilon_{ij}(\lambda)=1~\forall~i,j$. Broadly, this is the pixel-by-pixel sensitivity to incident photons. "Baked-in" is more than the detector quantum efficiency, it "bakes-in" the reduced efficiency due to any filter, any non-uniformity due to hardware or software amplification via gain modulations, as well as any other imperfections in the optical system that impact the ability for the detector to record photons incident on the primary mirror (all of that schmutz on our mirrors!).\\
-== Source Signal ==
-$I_{ij}(\lambda)$ is the source signal. It is measured in counts, and precision in the counts recorded come at the expense of signal contributions from the Dark and Bias signals (see below). The aim of any reduction pipeline is to take the signals in a raw light frame and reduce the contributions of other signals such that only this source signal is present. The result is called a science frame or science image and hypothetically is the closest thing to $I_{ij}(\lambda)$ we can muster with our system and reduction pipeline.\\
-== Thermal Signal ==
-$T_{ij}(\lambda)$ is the thermal signal, it is a component of the dark signal along with the bias (below). Thermal signal is a measure of the number electrons that accrued enough thermal energy to liberate themselves from the valence band of the CMOS semiconductor substrate and be read-off as signal by the Analog-Digital-Converter (ADU). As such, the thermal signal is time-dependent. I.e., the longer you give electrons in the valence band time to explore microstates made available by some heat bath, the more likely some will find the microstates that allow them to get from the valence band and into the conduction band to be read-off. Since the thermal signal is not generated by photons and the quantum efficiency of the pixels play no part, we can safely drop any implied $\lambda$ dependence from the thermal signal term. $T_{ij}(\lambda) \rightarrow T_{ij}$ \\
-== Integration Time ==
-$t$ is the exposure or integration time. "Integration" time is favored by the UMBC Obs Group because it reminds the reader that we are integrating radiance or flux over time. This integration also couples time with the counts generated by the source and thermal signals.\\
-== Bias Signal ==
-$B_{ij}(\lambda)$ is the bias signal, it is a component of the dark signal along with the thermal signal (above). Bias signal is a measure of the electrons that were liberated by the internal voltages that are constantly and non-uniformly applied across all pixels. These internal voltages are responsible for shuttling liberated electrons (either by photons or thermal excitation) into storage circuits until read-time by the ADC. These internal voltages are responsible for some initial 'jolt' of extra elections, but after this initial 'jolt', thermal electrons can continue to trickle into the storage circuits (these contribute to the thermal signal). In this way, the bias signal is not time-integrated. Further, since the bias signal is not generated by photons and the quantum efficiency of the pixels play no part, we can safely drop any implied $\lambda$ dependence from the bias signal term. $B_{ij}(\lambda) \rightarrow B_{ij}$\\
-==== Isolating Source Signal ====
-$$L_{ij}(\lambda) = \Bigl[\epsilon_{ij}(\lambda)I_{ij}(\lambda) + T_{ij}(\lambda)\Bigr]t + B_{ij}(\lambda)$$
-Removing spectral dependence
-$$L_{ij}(\lambda) = \Bigl[\epsilon_{ij}(\lambda)I_{ij}(\lambda) + T_{ij}\Bigr]t + B_{ij}$$
-Distribute integration time
-$$L_{ij}(\lambda) = \epsilon_{ij}(\lambda)I_{ij}(\lambda)t + T_{ij}t + B_{ij}$$
-Remove Bias Signal
-$$L_{ij}(\lambda) - B_{ij}  = \epsilon_{ij}(\lambda)I_{ij}(\lambda)t + T_{ij}t $$
-Remove Thermal Signal
-$$L_{ij}(\lambda) - B_{ij} - T_{ij}t = \epsilon_{ij}(\lambda)I_{ij}(\lambda)t $$
-Divide through by uniformity
-$$\frac{L_{ij}(\lambda) - B_{ij} - T_{ij}t}{\epsilon_{ij}(\lambda)} = I_{ij}(\lambda)t $$
-Finally arrive at science frame (Note that the integration time was left with the Source Signal)
-$$I_{ij}(\lambda)t = \frac{L_{ij}(\lambda) - B_{ij} - T_{ij}t}{\epsilon_{ij}(\lambda)}$$
-Some may choose to view the Dark Signal as one object -- the time-integrated Thermal Signal and constant Bias Signal
-$$I_{ij}(\lambda)t = \frac{L_{ij}(\lambda) - (B_{ij} + T_{ij}t)}{\epsilon_{ij}(\lambda)}$$
-$$I_{ij}(\lambda)t = \frac{L_{ij}(\lambda) - D_{ij}(\lambda)}{\epsilon_{ij}(\lambda)}$$
-=== Dark Signal ===
-In an effort to be super clear, the Dark Signal (and DARK frames as we'll discuss later) is built from the time-integrated thermal signal and the constant (but not uniform) bias signal. This is often glossed-over, but plays an important part in our data reduction pipeline as well as contributing to best practices for calibration frame collection.
-$$D_{ij}(\lambda) = B_{ij} + T_{ij}t$$
-=== How Many Calibration Frames Are Enough? ===
-Since each of the calibration frames mentioned above are fundamentally samplings of random processes, we need to ensure we're reasonably sampling these random processes. So we will take multiple calibration frames for each type. And given this, it'll be important that we find some measure of central tendency to effectively summarize the contribution of the various unwanted signals to the light frames.\\
-Light Signal: The likelihood that photons falls on a given pixel and generate a photoelectron is roughly Gaussian in pixel location around the projected center of the source and Poissonian in photoelectron generation.\\
-Thermal and Bias Signals: Each signal is Poissonian distributed with some Maxwell-Boltzmann Statistics sprinkled in.\\
-Uniformity: Also Poissonian distributed (recall from [[wiki:observational_astronomy:uniformity|Uniformity (FLATs)]] that FLAT frames are basically light frames of a uniformly illuminated field)\\
-For our purposes, I'll posit that 10-30 frames for each calibration frame is sufficient and choose further choose the MEDIAN as the measure of central tendency. #GraceHopper\\
-Refer to the Redux Pipeline and Unwanted Instrument Signals pages above for more information on each of these. And remember, removal of these unwanted signals from the system do not leave you with a science frame. We still have the additional sky intensity from light pollution as well as airmass extinction effects to account for.