User Tools

Site Tools


information_for_astronomers:user_guide:reduc_pointing

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
information_for_astronomers:user_guide:reduc_pointing [2013/10/21 13:07]
ubach [Toolbox Options]
information_for_astronomers:user_guide:reduc_pointing [2022/07/04 20:35] (current)
twedel
Line 3: Line 3:
 ===== Location of the Raw Data ===== ===== Location of the Raw Data =====
  
-The raw data of the 100m Effelsberg telescope is stored in MBFITS-Format. In Effelsberg the files are located in the directory /daten/Raw which should be available on every **Observer-PC**. Older data can be found in /daten/​Raw/​Raw-YYYY-MM. Most programs listed here support the flag "​fdir"​ to point the program to the directory where the MBFITS data can be found. Default is always /daten/Raw.+The raw data of the 100m Effelsberg telescope is stored in MBFITS-Format. In Effelsberg the files are located in the directory /daten/Raw which should be available on every **Observer-PC**. Older data can be found in /eff/data/​Raw/​Raw-YYYY-MM. /​eff/​data/​Raw is available on every **Observer-PC** and in the network in Bonn as well. Most programs listed here support the flag "​fdir"​ to point the program to the directory where the MBFITS data can be found. Default is always /daten/Raw.
  
-Every 30 minutes the raw data is synced to Bonn. It is accessible in /effbg/effdata/. More details on the location of data can be found here under [[Data storage and archive]]+Every 30 minutes the raw data is synced to Bonn. It is accessible in /eff/data/. Old data is accessible in /​hsm/​effarchive ​More details on the location of data can be found here under [[Data storage and archive]]
  
 +Current **Observer-PCs** are observer4 (64-bit) and observer7 (32-bit). Observers should use their MPIfR account to login to those machines, observes without an MPIfR account can use the common **obs2** account. ​
 ===== Inspecting scans by hand using the Toolbox ===== ===== Inspecting scans by hand using the Toolbox =====
  
-The MBFITS data can be inspected with any pogram ​that understand FITS Format e.g. "​fv"​ fits-viewer. You can look at the headers and tables and plot different data columns...+The MBFITS data can be inspected with any program ​that understand FITS Format e.g. "​fv"​ fits-viewer. You can look at the headers and tables and plot different data columns...
  
-However, most users might prefer a kind of pre-reduced view where you see the amplitude of the scan calibrated in units of the calibration temperature and with real arcseconds for the scanning axis. This is provided by the "​toolbox"​ program. ​It is currently in a development phase, but the latest stable version is running ​on all of the **observer-PCs** using user: **obs2**. Your normal MPI account ​should also work, the observer-PCs are connected ​to the /homes server, but you might need to adjust ​your PATH environment to get access to all programs.+However, most users might prefer a kind of pre-reduced view where you see the amplitude of the scan calibrated in units of the calibration temperature and with real arcseconds for the scanning axis. This is provided by the "​toolbox"​ program. ​The toolbox can be used on the **observer-PCs** ​(currently observer7 and observer4) ​using your normal MPI account. You only need to add /opt/​bin ​to your PATH variable. 
 + 
 +<​code>​ 
 +#bash 
 +export PATH=$PATH:/​opt/​bin 
 + 
 +#csh 
 +set path = ($path /opt/bin) 
 +</​code>​ 
 + 
 +Observers that don't have a MPI user Account can use the common **obs2** account.
  
 The toolbox can be used interactively by calling The toolbox can be used interactively by calling
Line 73: Line 84:
 |rfi=n| Will filter RFI peaks looking for peak increasing n times the rms.| |rfi=n| Will filter RFI peaks looking for peak increasing n times the rms.|
 |window=n| Defines the window size in multiples of the beam width for the second iteration of the Gaussian fit (orange box in the plot window). Default is 1.22| |window=n| Defines the window size in multiples of the beam width for the second iteration of the Gaussian fit (orange box in the plot window). Default is 1.22|
 +|rms| Prints the rms of the scan after subtracting the fit.|
 ^mapping options:| For viewing maps or converting them to FITS or nod2 standards.| ^mapping options:| For viewing maps or converting them to FITS or nod2 standards.|
 |color| Shows color plots instead of black and white.| |color| Shows color plots instead of black and white.|
Line 87: Line 99:
 ===== Observing Logs ===== ===== Observing Logs =====
  
-A program logging ​number of useful information is running on the observer PCIf you are observing by your self you can just send the log to your email address or you can ask the operator to do so.  +All observations are logged in MySQL data baseTo see the content ​of  
- +the data base, call the program
-If you don't have that log there are a number ​of ways to obtain an observing log from previous dates. ​ +
- +
-===ObsLog.py=== +
-In **/​daten/​ObsLogs** there are log-files for every observing day starting from 12 UT noon on the previous day. The files are named for each date YYMMDD.prot. A small script that can be used to search through these files is called **ObsLog.py**+
 <​code>​ <​code>​
- ​Usage:​ ObsLog.py [date=YYMMDD] or [fromto=YYMMDD,​YYMMDD] and/or [scan=from#,​to#​] +obslog
- +
-ObsLog.py date= and fromto= work without scan=, but scan= not without one of the previous+
 </​code>​ </​code>​
-It can be used to print out a specific day, many days, and restrict the scan number rangeUsing only scan numbers might be misleading because scan number repeat after 9999. ObsLog.py is the fastest way to produce a log, since it only searches through some ascii files.+on observer2For more information see  
 +[[information_for_astronomers:​user_guide:​pyobslog|Obslogger]].
  
-For example: +Once you have found your observations in the Obslogger, the log can be saved in a text file that can be used as an input file for the further data reduction below. 
-<​code>​ +===== Reducing a number of scans at once =====
-obs2@observer3:​~$ ObsLog.py date=110418 +
-# 1    2         ​3 ​     4            5    6      7      8     ​9 ​   10    11   ​12 ​    ​13 ​  ​14 ​    ​15 ​   16    17  18    19   ​20 ​   21  22  23 24 +
-#Scan Date       ​UTC ​ Source ​       Azm   ​Elv ​  ​Freq ​  ​col* ​ Nule Nula  X-lin Y-lin Z-lin X-rot Y-rot Opos  Temp Hum  Press Ref   ​wv ​ wd  ts es +
-.... +
-2684 2011-04-18 01:13 3C286         236.9 61.0  24.00  4.3   ​2.1 ​  ​0.0 ​  ​0.0 ​ 27.0  -7.8  28.0   ​0.0 ​  ​0.0 ​ 3.7 70.0 984.6 62.0  0.9  98  4  4 +
-2685 2011-04-18 01:15 3C286         237.8 60.7  24.00  2.6  -1.9   ​0.0 ​  ​0.0 ​ 27.0  -7.8  28.0   ​0.0 ​  ​0.0 ​ 3.7 70.0 984.6 62.0  0.3 157  2  2 +
-2686 2011-04-18 01:17 3C286         238.2 60.5  24.00  2.6  -1.9   ​0.0 ​  ​0.0 ​ 27.0  -7.3  28.0   ​0.0 ​  ​0.0 ​ 3.7 70.0 984.6 62.0 -0.2 208  4  4 +
-2687 2011-04-18 01:25 W3MAIN ​       369.2 23.5  23.41 -0.0  -0.5   ​0.0 ​  ​0.0 ​ 27.0  -7.3  28.0   ​0.0 ​  ​0.0 ​ 3.7 70.0 984.1 62.0  0.0 158  2  2 +
-2688 2011-04-18 01:35 3C345         131.9 75.4  23.41 -0.0  -0.5   ​0.0 ​  ​0.0 ​ 27.0  -7.3  28.0   ​0.0 ​  ​0.0 ​ 3.5 72.0 984.1 63.0 -0.0 164  4  4 +
-2689 2011-04-18 01:38 3C345         133.5 75.7  23.41  1.2   ​7.7 ​  ​0.0 ​  ​0.0 ​ 27.0  -7.3  28.0   ​0.0 ​  ​0.0 ​ 3.5 71.0 984.1 62.0 -0.0 191  2  2 +
-2690 2011-04-18 01:39 3C345         134.2 75.8  23.41  1.2   ​7.7 ​  ​0.0 ​  ​0.0 ​ 27.0  -7.6  28.0   ​0.0 ​  ​0.0 ​ 3.5 71.0 984.1 63.0 -0.0 238  4  4 +
-2691 2011-04-18 01:42 3C345         136.1 76.2  25.06  0.4   ​6.1 ​  ​0.0 ​  ​0.0 ​ 27.0  -7.5  28.0   ​0.0 ​  ​0.0 ​ 3.6 71.0 984.1 62.0  0.4 230  4  4 +
-2692 2011-04-18 01:45 3C345         137.8 76.4  25.06 -0.1   ​7.3 ​  ​0.0 ​  ​0.0 ​ 27.0  -7.5  28.0   ​0.0 ​  ​0.0 ​ 3.6 71.0 984.1 62.0  0.5 232  2  2 +
-.... +
-</​code>​+
  
-===Logbook.py===+Looking at single scan might be appropriate for checking the data during an observation,​ but for calibration or flux density monitoring a more automatic way is preferred. There is a collection of scripts and programs that can be used to perform all the tasks to obtain flux density calibrated data. The scripts are located in /opt/bin on the **Observer-PCs**.
  
-Logbook.py lists by default the last 1000 scans. This might take some time, because Logbook.py is reading the original MBFITS data files to get the information. Using +==== Raw Data Processing ====
-<​code>​ +
-Logbook.py 100 +
-</​code>​ +
-it lists only the last 100 scans. The time range can be restricted by the option +
-<​code>​ +
-Logbook.py 10000 tstart=YYYY-MM-DD tstop=YYYY-MM-DD +
-</​code>​ +
-where "​tstart"​ marks the more recent time (if not specified the print out starts today) and "​tstop"​ specifies the stop time in the past. The 10000 is just chosen to be big enough to cover most time ranges. Logbook.py can be used only to produce logs from more recent data because it searches only in /daten/Raw.+
  
-For example: +The scripts and programs mostly use a file that contains all the scan numbers to be reduced. The scripts **dblog2scan.py** or **dblog2scan_wea.py** produce such lists from observing logs written Obslogger
-<​code>​ + 
-obs2@observer3:​~$ Logbook.py tstart=2011-04-18 tstop=2011-04-18 +In dblog2scan.py one can optionally restrict the frequency by the receiver version and frequency in GHz, if the log contains entries at different frequencies.
-# SCAN   ​SUB ​  ​OBJECT ​        ​SCANTYPE ​   FEBE            PROJECT ​     DATE_OBS ​             MJD +
-.... +
-  ​2692 ​   2    3C345           ​FOCUS ​     P13mm-PBE ​      ​46-09 ​       2011-04-18T01:​45:​17 ​  ​55669.0731 +
-  ​2691 ​   4    3C345           ​POINT ​     P13mm-PBE ​      ​46-09 ​       2011-04-18T01:​42:​56 ​  55669.0715 +
-  2690    4    3C345           ​POINT ​     P13mm-PBE ​      ​46-09 ​       2011-04-18T01:​40:​05 ​  ​55669.0695 +
-  2689    2    3C345           ​FOCUS ​     P13mm-PBE ​      ​46-09 ​       2011-04-18T01:​38:​44 ​  ​55669.0686 +
-  2688    4    3C345           ​POINT ​     P13mm-PBE ​      ​46-09 ​       2011-04-18T01:​36:​22 ​  ​55669.0669 +
-  2687    2    W3MAIN ​         ONOFF      P13mm-FFTS ​     46-09        2011-04-18T01:​26:​08 ​  ​55669.0598 +
-  2686    4    3C286           ​POINT ​     P13mm-PBE ​      ​46-09 ​       2011-04-18T01:​17:​28 ​  ​55669.0538 +
-  2685    2    3C286           ​FOCUS ​     P13mm-PBE ​      ​46-09 ​       2011-04-18T01:​16:​08 ​  ​55669.0529 +
-  2684    4    3C286           ​POINT ​     P13mm-PBE ​      ​46-09 ​       2011-04-18T01:​13:​45 ​  ​55669.0512 +
-.... +
-</​code>​ +
-===log.py===+
  
-log.py is a small script that prints some more detailed information than Logbook.py. 
 <​code>​ <​code>​
- Usagelog.py scan1 scan2 [fdir=<​PATH>​]+ubach@observer4:~$ dblog2scan_wea.py 
  
- Options: + Usagedblog2scan.py log-file [receiver] [freq GHz]
- ​fdir=<​PATH>:​ if the data are not in /daten/Raw any more +
-</​code>​ +
- +
-It can be also used to produce a log from older data in the archival directories,​ /​datan/​Raw/​Raw-YYYY-MM specifying "​fdir"​. +
- +
-For example: +
-<​code>​ +
-obs2@observer3:​~$ ​log.py 2684 2692 +
-# ***  Date: 2011-04-18  Project-ID: 46-09     ​Observer-Operator:​ ALEX       *** +
-# **  Temp =   3.7 deg C     Humid = 70.0 %     Press =  984.6 hPa  +
-#Scan Sub Source ​        ​UT ​      ​LST ​   FE-BE        Mode       ​Freq ​       Type  Sw-mode AZI  ELV   ​Col* ​ NULE  z-Lin +
- ​2684 ​ 4  3C286        01:13:45 15:23:59 P13mm-PBE ​   CONTINUUM ​ 24000.00 ​   POINT   TOTP 236.9 61.0   ​4.3 ​  ​2.1 ​ -7.8 +
- ​2685 ​ 2  3C286        01:16:08 15:26:36 P13mm-PBE ​   CONTINUUM ​ 24000.00 ​   FOCUS   TOTP 237.8 60.7   ​2.6 ​ -1.9  -7.8 +
- ​2686 ​ 4  3C286        01:17:28 15:27:44 P13mm-PBE ​   CONTINUUM ​ 24000.00 ​   POINT   TOTP 238.2 60.5   ​2.6 ​ -1.9  -7.3 +
- ​2687 ​ 2  W3MAIN ​      ​01:​26:​08 15:36:25 P13mm-FFTS ​  ​SPECTROSCO 23407.18 ​   ONOFF   TOTP 369.2 23.5  -0.0  -0.5  -7.3 +
- ​2688 ​ 4  3C345        01:36:22 15:46:40 P13mm-PBE ​   CONTINUUM ​ 23407.18 ​   POINT   TOTP 131.9 75.4  -0.0  -0.5  -7.3 +
- ​2689 ​ 2  3C345        01:38:44 15:49:18 P13mm-PBE ​   CONTINUUM ​ 23407.18 ​   FOCUS   TOTP 133.4 75.7   ​1.2 ​  ​7.7 ​ -7.3 +
- ​2690 ​ 4  3C345        01:40:05 15:50:24 P13mm-PBE ​   CONTINUUM ​ 23407.18 ​   POINT   TOTP 134.2 75.8   ​1.2 ​  ​7.7 ​ -7.6 +
- ​2691 ​ 4  3C345        01:42:56 15:53:15 P13mm-PBE ​   CONTINUUM ​ 25056.25 ​   POINT   TOTP 136.0 76.1   ​0.4 ​  ​6.1 ​ -7.5 +
- ​2692 ​ 2  3C345        01:45:17 15:55:56 P13mm-PBE ​   CONTINUUM ​ 25056.25 ​   FOCUS   TOTP 137.7 76.4  -0.1   ​7.3 ​ -7.5+
  
 +Give a frontend designation (e.g. 28.1) and additional a frequency
 </​code>​ </​code>​
  
-===== Reducing a number of scans at once ===== +**dblog2scan.py** will just produce the files scans (input file for weather.py and corr_point.py) ​and scanlist (to be attached ​to the reduce.par file), **dblog2scan_wea.py** will produce ​an additional weather.dat file with weather information for each scanThe weather file is useful for the opacity correction described below.
- +
-Looking at single scan might be appropriate ​for checking the data during an observation,​ but for calibration or flux density monitoring a more automatic way is preferredThere is a collection of scripts ​and programs that can be used to perform all the tasks to obtain flux density calibrated dataThe scripts are located in /​home/​obs2/​bin on the observer3-PC. +
- +
-==== Raw Data Processing ==== +
- +
-The scripts and programs mostly use a file that contains all the scan numbers to be reduced. The scripts ​**log2scan.py** or **logbook2scan.py** produce ​such lists from observing logs written by log.py or Logbook.py, respectively.  +
- +
-In log2scan.py one can optionally restrict ​the frequency if the log contains entries at different frequencies. +
-<​code>​ +
-log2scan.py +
- ​Usage:​ log2scan.py log-file [freq GHz] +
-</​code>​+
  
 ---- ----
 +\\
 The script **reduce.py** can be used to reduce a number of given scans using the toolbox with a list of options. Some example parameter files are stored in /​home/​obs2/​flux_monit/​reduce-par. E.g., reduc28.par for the 2.8cm SFK receiver. Calling just reduce.py prints out some help as well. The script **reduce.py** can be used to reduce a number of given scans using the toolbox with a list of options. Some example parameter files are stored in /​home/​obs2/​flux_monit/​reduce-par. E.g., reduc28.par for the 2.8cm SFK receiver. Calling just reduce.py prints out some help as well.
  
 <​code>​ <​code>​
-obs2@observer3:~$ reduce.py ​+obs2@observer9:~$ reduce.py ​
  
  ​Usage:​ reduce.py par-file  ​Usage:​ reduce.py par-file
Line 258: Line 196:
 Since the amplitudes from the toolbox are in units of the Tcal one just have to multiply the given numbers by the Tcal to get the correct system temperature. The reduce.py procedure provides an all.fit file that contains all the single fit-files. The further programs work with this all.fit. Since the amplitudes from the toolbox are in units of the Tcal one just have to multiply the given numbers by the Tcal to get the correct system temperature. The reduce.py procedure provides an all.fit file that contains all the single fit-files. The further programs work with this all.fit.
  
-The script **weather.py** reads the weather information ​for the MBFITS filesmultiplies the Tsys found in the all.fit by a given Tcal and prints out a weather.dat and a LIST.tsys, with the opacity information for each scan that is listed in the scanlist-file. The minimum zenith opacity is fitted from the Tsys vs. airmass distribution and is presented in the plot Opacities.eps. Weather.eps contains the weather info.+The script **weather.py** reads the weather information ​from the weather.dat file or directly from the MBFITS files. It multiplies the Tsys found in the all.fit by a given Tcal and prints out a LIST.tsys, with the opacity information for each scan that is listed in the scanlist-file. When the weather data is read from the raw MBFITS file the script will write a new weather.dat for alter use. The minimum zenith opacity is fitted from the Tsys vs. airmass distribution and is presented in the plot Opacities.eps. Weather.eps contains the weather info.
  
 <​code>​ <​code>​
-obs2@observer3:~$ weather.py ​+obs2@observer9:~$ weather.py ​
  Task to read weather data from MBFITS files,  Task to read weather data from MBFITS files,
  save them in ASCII format, and compute LIST.tsys.  save them in ASCII format, and compute LIST.tsys.
Line 289: Line 227:
  
 <​code>​ <​code>​
-obs2@observer3:~$ corr_point.py ​+obs2@observer9:~$ corr_point.py ​
    
  Task to reduce fit-files from Peters Toolbox  Task to reduce fit-files from Peters Toolbox
Line 307: Line 245:
 </​code>​ </​code>​
  
-For example the Tcal at 2.8cm is 7.5 K and scan numbers are stored in scanlist+For example the Tcal at 2.8cm is 7.5 K and scan numbers are stored in the file scans.
  
 <​code>​ <​code>​
-obs2@observer3:​~/​ubach/​measurements/​2011_03_16_poi/​2.8cm/​0.Raw$ corr_point.py ​scanlist ​7.5+obs2@observer3:​~/​ubach/​measurements/​2011_03_16_poi/​2.8cm/​0.Raw$ corr_point.py ​scans 7.5
  
  ​****************************************************  ​****************************************************
information_for_astronomers/user_guide/reduc_pointing.1382353629.txt.gz · Last modified: 2013/10/21 13:07 by ubach