User Tools

Site Tools


information_for_astronomers:user_guide:reduction_of_spectroscopic_measurements

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
information_for_astronomers:user_guide:reduction_of_spectroscopic_measurements [2014/11/30 14:30]
bwinkel [Interactive mode - OfflinePipeline]
information_for_astronomers:user_guide:reduction_of_spectroscopic_measurements [2018/12/12 16:18] (current)
akraus
Line 1: Line 1:
-===== Reducing spectral line data =====+====== Reducing spectral line data ======
  
 The raw data (in MBFITS format) are converted to CLASS format automatically. This is done using the method described in Winkel, Kraus, & Bach, //Unbiased flux calibration methods for spectral-line radio observations//,​ [[http://​adsabs.harvard.edu/​abs/​2012arXiv1203.0741W| A&A 540, A140, 2012]] (Section 3.4 and Section 4.5) - assuming a //constant system temperature//​ (Tsys) and //flat calibration signal// (Tcal). The latter is produced by a noise diode the power of which is in most cases directly fed into the waveguide within the receiver. //​Neglecting the frequency-dependence of Tsys and Tcal introduces a bias, especially for wide-band observations!//​ (For more information see Winkel, Kraus, & Bach 2012 where we also show ways to do it better.) The raw data (in MBFITS format) are converted to CLASS format automatically. This is done using the method described in Winkel, Kraus, & Bach, //Unbiased flux calibration methods for spectral-line radio observations//,​ [[http://​adsabs.harvard.edu/​abs/​2012arXiv1203.0741W| A&A 540, A140, 2012]] (Section 3.4 and Section 4.5) - assuming a //constant system temperature//​ (Tsys) and //flat calibration signal// (Tcal). The latter is produced by a noise diode the power of which is in most cases directly fed into the waveguide within the receiver. //​Neglecting the frequency-dependence of Tsys and Tcal introduces a bias, especially for wide-band observations!//​ (For more information see Winkel, Kraus, & Bach 2012 where we also show ways to do it better.)
  
-The data files can be found on the //​observer4//​ or //observer3// in the directory +The data files can be found on the //​observer4//​ or //observer6// in the directory 
-**/​daten/​Class/​class_YYYY_MM_DD.100m**. There might be a short delay until the data is processed and is visible in the class file after the observation has finished+**/​daten/​Class/​[fw_]class_YYYY_MM_DD.100m**. There might be a short delay until the data is processed and is visible in the class file after the observation has finished
  
 **Spectra in Class are always in units of Tcal!** To obtain Kelvins or Jy/Beam you have to multiply with the temperature of the noise diode (Tcal) and appropriate antenna sensitivities/​efficiencies. Approximate values for this can be found on our receiver website [[http://​www3.mpifr-bonn.mpg.de/​div/​effelsberg/​receivers/​receiver.html|receiver website]]. However, it is highly recommended to do calibration measurements during your observation for better accuracy. Furthermore,​ the automatic Class pipeline does not account for atmospheric opacity, elevation-gain curve, and antenna efficiency. Information about these steps can be found in Kraus 2009 (calibration memo). **Spectra in Class are always in units of Tcal!** To obtain Kelvins or Jy/Beam you have to multiply with the temperature of the noise diode (Tcal) and appropriate antenna sensitivities/​efficiencies. Approximate values for this can be found on our receiver website [[http://​www3.mpifr-bonn.mpg.de/​div/​effelsberg/​receivers/​receiver.html|receiver website]]. However, it is highly recommended to do calibration measurements during your observation for better accuracy. Furthermore,​ the automatic Class pipeline does not account for atmospheric opacity, elevation-gain curve, and antenna efficiency. Information about these steps can be found in Kraus 2009 (calibration memo).
Line 11: Line 11:
  
 ===== Using class ===== ===== Using class =====
-To reduce your (spectroscopic and pointing) data with ''​class''​ one should use the **observer3** or **observer4** computer. ​Everyone ​with an account in the mpifr network can login to this computer with his/her account. It is connected to the ''/​homes''​ server. It is also connected to ''/​homes/​astro/​gag'',​ so your gildas package of choice should work. It is also possible to use a local version of class, which is used to write the spectroscopic and pointing data. To enable this, change to the bash shell by typing bash and source the init file:\\ +To reduce your (spectroscopic and pointing) data with ''​class''​ one should use the **observer4** or **observer6** computer. ​Both are in the MPIfR LDAP environment,​ i.e., everyone ​with an account in the MPIfR network can login to this computer with his/her account. It is connected to the ''/​homes''​ server. It is also connected to ''/​soft/​astro/​gag'',​ so your gildas package of choice should work. It is also possible to use the Effelsberg ​version of class that is used to write the spectroscopic and pointing data. To enable this (if you are using a C-Shell like tcsh), change to the bash shell by typing ​ 
-<code bash>​source /opt/EffelsbergPipeline/init_classwriter.sh</​code>​+<​code>​bash</​code>​  
 +and source the init file:\\ 
 +<code bash>​source /opt/bin/init_rtsoft.sh</​code>​
 afterward you can run afterward you can run
 <code bash>​class</​code>​ <code bash>​class</​code>​
Line 46: Line 48:
  
 ===== Reprocessing spectra using the Class pipeline ===== ===== Reprocessing spectra using the Class pipeline =====
-Sometimes it may be necessary to re-reduce the data with the Class pipeline. This can be done on the observer4 computer. The necessary software is installed in ''/​opt/​EffelsbergSpecPipeline''​. There is a bash script to set all the necessary paths. "​tcsh"​ shell users should change to "​bash"​ by typing ''​bash'' ​(the teleskop and obs2 account are already bash accounts). Launch the following command:+Sometimes it may be necessary to re-reduce the data with the Class pipeline. This can be done on the observer4 computer. The necessary software is installed in ''/​opt/​specpipeline''​. There is a bash script to set all the necessary paths. "​tcsh"​ shell users should change to "​bash"​ by typing ''​bash''​. Launch the following command:
  
-<code bash>​source /opt/EffelsbergSpecPipeline/init_classwriter.sh</​code>​+<code bash> 
 +source /opt/bin/init_rtsoft.sh 
 +</​code>​
  
-Now you should be ready to run the offline Class calibration pipeline. There are two ways to do, an interactive tool and a command-line tool (for simple tasks).+Now you should be ready to run the offline Class calibration pipeline. There are two ways to go, an interactive tool and a command-line tool (for simple tasks).
  
 ==== Interactive mode - OfflinePipeline ==== ==== Interactive mode - OfflinePipeline ====
Line 85: Line 89:
 scannums = range(6901, 6930) scannums = range(6901, 6930)
  
-# alternatively,​ you can also use a shell-glob pattern and do some python ​magic to obtain associated scan numbers+# alternatively,​ you can also use a shell-glob pattern and do some rexexp ​magic to obtain associated scan numbers
 import glob, re import glob, re
 filenames = sorted(glob.glob('/​daten/​Raw/​*_27-13_*.FITS'​)) filenames = sorted(glob.glob('/​daten/​Raw/​*_27-13_*.FITS'​))
Line 110: Line 114:
 |baseband|"​*"​|Process only data with specified baseband number (e.g., 1), "​*"​ means all| |baseband|"​*"​|Process only data with specified baseband number (e.g., 1), "​*"​ means all|
 |fswFold|True|The pipeline does the frequency switch folding operation (you really want this and not let Class do it). See [[information_for_astronomers:​fsw_bug|About Bandpass Ghosts]].| |fswFold|True|The pipeline does the frequency switch folding operation (you really want this and not let Class do it). See [[information_for_astronomers:​fsw_bug|About Bandpass Ghosts]].|
-|regridMethod|"​fft"​|TODO|+|regridMethod|"​fft"​|Which regrid algorithm to use for frequency shifting. The default ('​fft'​) algorithm is fast and is the only which doesn'​t lower spectral resolution. **However, ringing effects might occur.** This is because the FFTS spectra are not fully-sampled in the strict sense of Shannon(-Nyquist)s sampling theorem. Ringing usually occurs when strong narrowband features are present. Other methods include '​gaussian'​ (for a convolution based approach) and [[http://​docs.scipy.org/​doc/​scipy-0.14.0/​reference/​generated/​scipy.interpolate.interp1d.html#​scipy.interpolate.interp1d|scipy'​s]] regrid algorithms: '​linear',​ '​nearest',​ '​zero',​ '​slinear',​ '​quadratic',​ '​cubic'​.|
 |averMethod|"​mean"​|Allows to specifiy the averaging-method ('​mean'​ or '​median',​ the latter might help with RFI)| |averMethod|"​mean"​|Allows to specifiy the averaging-method ('​mean'​ or '​median',​ the latter might help with RFI)|
 |skipDumps|(2,​ 0)|Allows to skip dumps at beginning/​end of a subscan. At the beginning, the telescope sometimes is not yet fully on-source.| |skipDumps|(2,​ 0)|Allows to skip dumps at beginning/​end of a subscan. At the beginning, the telescope sometimes is not yet fully on-source.|
Line 116: Line 120:
 |shiftFFTSRefChannel|False|Before February 2011 the reference channel of the FFTS was wrong (in Fits header) and has to be shifted by +0.5 channels if the spectrum was not flipped, otherwise -0.5 channels. See also the [[fix:​frequency_shift_in_ffts_observation_due_to_wrong_reference_channel|News section]].| |shiftFFTSRefChannel|False|Before February 2011 the reference channel of the FFTS was wrong (in Fits header) and has to be shifted by +0.5 channels if the spectrum was not flipped, otherwise -0.5 channels. See also the [[fix:​frequency_shift_in_ffts_observation_due_to_wrong_reference_channel|News section]].|
 |autoLsrCorr|True|Per default the class pipeline attempts to correct the data for LSR Doppler shifts (due observer motion w.r.t. the LSR) if not already accounted for by hardware (LO tuning). If you really wanted a fixed-frequency setup (e.g., for system testing) you can disable automatic correction with this keyword.| |autoLsrCorr|True|Per default the class pipeline attempts to correct the data for LSR Doppler shifts (due observer motion w.r.t. the LSR) if not already accounted for by hardware (LO tuning). If you really wanted a fixed-frequency setup (e.g., for system testing) you can disable automatic correction with this keyword.|
 +|doAatm|False|Apply AATM atmospheric model opacity correction (based on weather data).|
 +|gainCurve|(1.,​ 0., 0.)|Apply gain curve correction (must be a tuple of (a0, a1, a2); see [[information_for_astronomers:​rx_list|receiver details]].|
 +|tcal|1.|Apply tcal factor. Note, this can also be used to scale the data (e.g., to produce Jy/Beam instead of K, or to apply antenna efficiency). If you want to use different tcal factors per Baseband or FeBe you will need to use the baseband/​febe arguments and run the pipeline several times (once for each baseband/​febe).|
  
  
 +==== Shell processing - CmdLinePipeline ====
 +Often one just wants to reprocess a bunch of Fitsfiles. For this, we made a small shell-(python-)script.
 +
 +<code bash>
 +python CmdLinePipeline.py -h
 +</​code>​
 +<code txt>
 +usage: CmdLinePipeline.py [-h] [-c CLASSFILE] [--baseband BASEBAND]
 +                          [--febe FEBE] [--no-fsw-fold]
 +                          [--no-auto-lsr-correction]
 +                          [--regrid-method {fft,​gaussian,​linear,​nearest,​zero,​slinear,​quadratic,​cubic}]
 +                          [--averaging-method {mean,​median}]
 +                          [--skip-dumps SKIPDUMPS SKIPDUMPS] [--spectral-flip]
 +                          [--shift-ref-chan]
 +                          fitsnames [fitsnames ...]
 +
 +Command line interface to Class pipeline
 +
 +positional arguments:
 +  fitsnames ​            list of raw-fits files
 +
 +optional arguments:
 +  -h, --help ​           show this help message and exit
 +  -c CLASSFILE, --classfile CLASSFILE
 +                        name of output class file (default
 +                        "<​date>​_<​project>​.100m"​) you can choose any name,
 +                        <​date>​ and <​project>​ are meta-variables and will be
 +                        replaced by what is found in the fits files
 +                        (potentially creating more than one class file)
 +  --baseband BASEBAND ​  ​Filter by baseband
 +  --febe FEBE           ​Filter by febe
 +  --no-fsw-fold ​        do fswitch fold in pipeline (rather than class)
 +  --aatm ​               apply AATM opacity correction
 +  --gaincurve a0 a1 a2  apply gain curve correction (default: 1., 0., 0.)
 +                        Note, early versions of python-argparse dont support
 +                        scientific notation
 +  --tcal TCAL           apply tcal factor (default: 1.) Note, this can also be
 +                        used to scale the data (e.g., to produce Jy/Beam
 +                        instead of K, or to apply antenna efficiency).If you
 +                        want to use different tcal factors per baseband or
 +                        FeBe you will need to use the --baseband/​--febe
 +                        arguments and run the pipeline several times (once for
 +                        each baseband/​febe)
 +  --no-auto-lsr-correction
 +                        do not automatically correct for LSR shifts in fixed-
 +                        freq mode, default: False
 +  --regrid-method {fft,​gaussian,​linear,​nearest,​zero,​slinear,​quadratic,​cubic}
 +                        which method to use for fswitch freq shifting
 +                        (default: fft) possible regridMethods are: fft,
 +                        linear, gaussian (the latter is recommended if you
 +                        experience ringing in the spectrum, linear and
 +                        gaussian decrease spectral resolution!)
 +  --averaging-method {mean,​median}
 +                        averaging method (mean or median, default: mean)
 +  --skip-dumps SKIPDUMPS SKIPDUMPS
 +                        dumps to skip at beginning/​end (tuple), default: 2 0
 +  --spectral-flip ​      ​additional spectral flip of the data arrays, default:
 +                        False
 +  --shift-ref-chan ​     before February 2011 the reference channel of the FFTS
 +                        had to beshifted by +0.5 channels if the spectrum was
 +                        not IF flipped, otherwise -0.5, default: False
 +
 +</​code>​
 +
 +The meaning of the parameters is like in the Table above (just slightly different argument names). The only difference is the '-c CLASSFILE'​ option that allows to give a certain class name. Default is "<​date>​_<​project>​.100m"​ where the date and project code are automatically filled-in from what is found in the MBfits files (this can lead to multiple class files obviously).
 +
 +Typical usage would be:
 +<code bash>
 +# put everything into one class file
 +python CmdLinePipeline.py -c '/​homes/​user/​myclass.100m'​ /​daten/​Raw/​20141201*myprojectcode*XFFTS*
 +
 +# put everything into separate class files (split by observing date and project code)
 +python CmdLinePipeline.py -c '/​homes/​user/<​date>​_<​project>​.100m'​ /​daten/​Raw/​20141201*myprojectcode*XFFTS*
 +
 +# process just one MBFITS file, and put class file(s) into current directory
 +python CmdLinePipeline.py /​daten/​Raw/​20141211_2506_62-14_S13mm-XFFTS.FITS
 +</​code>​
information_for_astronomers/user_guide/reduction_of_spectroscopic_measurements.1417354219.txt.gz · Last modified: 2014/11/30 14:30 by bwinkel