This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision | ||
information_for_astronomers:user_guide:reduc_pointing [2013/10/21 13:07] ubach [Toolbox Options] |
information_for_astronomers:user_guide:reduc_pointing [2022/07/04 20:35] (current) twedel |
||
---|---|---|---|
Line 3: | Line 3: | ||
===== Location of the Raw Data ===== | ===== Location of the Raw Data ===== | ||
- | The raw data of the 100m Effelsberg telescope is stored in MBFITS-Format. In Effelsberg the files are located in the directory /daten/Raw which should be available on every **Observer-PC**. Older data can be found in /daten/Raw/Raw-YYYY-MM. Most programs listed here support the flag "fdir" to point the program to the directory where the MBFITS data can be found. Default is always /daten/Raw. | + | The raw data of the 100m Effelsberg telescope is stored in MBFITS-Format. In Effelsberg the files are located in the directory /daten/Raw which should be available on every **Observer-PC**. Older data can be found in /eff/data/Raw/Raw-YYYY-MM. /eff/data/Raw is available on every **Observer-PC** and in the network in Bonn as well. Most programs listed here support the flag "fdir" to point the program to the directory where the MBFITS data can be found. Default is always /daten/Raw. |
- | Every 30 minutes the raw data is synced to Bonn. It is accessible in /effbg/effdata/. More details on the location of data can be found here under [[Data storage and archive]] | + | Every 30 minutes the raw data is synced to Bonn. It is accessible in /eff/data/. Old data is accessible in /hsm/effarchive More details on the location of data can be found here under [[Data storage and archive]] |
+ | Current **Observer-PCs** are observer4 (64-bit) and observer7 (32-bit). Observers should use their MPIfR account to login to those machines, observes without an MPIfR account can use the common **obs2** account. | ||
===== Inspecting scans by hand using the Toolbox ===== | ===== Inspecting scans by hand using the Toolbox ===== | ||
- | The MBFITS data can be inspected with any pogram that understand FITS Format e.g. "fv" fits-viewer. You can look at the headers and tables and plot different data columns... | + | The MBFITS data can be inspected with any program that understand FITS Format e.g. "fv" fits-viewer. You can look at the headers and tables and plot different data columns... |
- | However, most users might prefer a kind of pre-reduced view where you see the amplitude of the scan calibrated in units of the calibration temperature and with real arcseconds for the scanning axis. This is provided by the "toolbox" program. It is currently in a development phase, but the latest stable version is running on all of the **observer-PCs** using user: **obs2**. Your normal MPI account should also work, the observer-PCs are connected to the /homes server, but you might need to adjust your PATH environment to get access to all programs. | + | However, most users might prefer a kind of pre-reduced view where you see the amplitude of the scan calibrated in units of the calibration temperature and with real arcseconds for the scanning axis. This is provided by the "toolbox" program. The toolbox can be used on the **observer-PCs** (currently observer7 and observer4) using your normal MPI account. You only need to add /opt/bin to your PATH variable. |
+ | |||
+ | <code> | ||
+ | #bash | ||
+ | export PATH=$PATH:/opt/bin | ||
+ | |||
+ | #csh | ||
+ | set path = ($path /opt/bin) | ||
+ | </code> | ||
+ | |||
+ | Observers that don't have a MPI user Account can use the common **obs2** account. | ||
The toolbox can be used interactively by calling | The toolbox can be used interactively by calling | ||
Line 73: | Line 84: | ||
|rfi=n| Will filter RFI peaks looking for peak increasing n times the rms.| | |rfi=n| Will filter RFI peaks looking for peak increasing n times the rms.| | ||
|window=n| Defines the window size in multiples of the beam width for the second iteration of the Gaussian fit (orange box in the plot window). Default is 1.22| | |window=n| Defines the window size in multiples of the beam width for the second iteration of the Gaussian fit (orange box in the plot window). Default is 1.22| | ||
+ | |rms| Prints the rms of the scan after subtracting the fit.| | ||
^mapping options:| For viewing maps or converting them to FITS or nod2 standards.| | ^mapping options:| For viewing maps or converting them to FITS or nod2 standards.| | ||
|color| Shows color plots instead of black and white.| | |color| Shows color plots instead of black and white.| | ||
Line 87: | Line 99: | ||
===== Observing Logs ===== | ===== Observing Logs ===== | ||
- | A program logging a number of useful information is running on the observer PC. If you are observing by your self you can just send the log to your email address or you can ask the operator to do so. | + | All observations are logged in a MySQL data base. To see the content of |
- | + | the data base, call the program | |
- | If you don't have that log there are a number of ways to obtain an observing log from previous dates. | + | |
- | + | ||
- | ===ObsLog.py=== | + | |
- | In **/daten/ObsLogs** there are log-files for every observing day starting from 12 UT noon on the previous day. The files are named for each date YYMMDD.prot. A small script that can be used to search through these files is called **ObsLog.py** | + | |
<code> | <code> | ||
- | Usage: ObsLog.py [date=YYMMDD] or [fromto=YYMMDD,YYMMDD] and/or [scan=from#,to#] | + | obslog |
- | + | ||
- | ObsLog.py date= and fromto= work without scan=, but scan= not without one of the previous | + | |
</code> | </code> | ||
- | It can be used to print out a specific day, many days, and restrict the scan number range. Using only scan numbers might be misleading because scan number repeat after 9999. ObsLog.py is the fastest way to produce a log, since it only searches through some ascii files. | + | on observer2. For more information see |
+ | [[information_for_astronomers:user_guide:pyobslog|Obslogger]]. | ||
- | For example: | + | Once you have found your observations in the Obslogger, the log can be saved in a text file that can be used as an input file for the further data reduction below. |
- | <code> | + | ===== Reducing a number of scans at once ===== |
- | obs2@observer3:~$ ObsLog.py date=110418 | + | |
- | # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 | + | |
- | #Scan Date UTC Source Azm Elv Freq col* Nule Nula X-lin Y-lin Z-lin X-rot Y-rot Opos Temp Hum Press Ref wv wd ts es | + | |
- | .... | + | |
- | 2684 2011-04-18 01:13 3C286 236.9 61.0 24.00 4.3 2.1 0.0 0.0 27.0 -7.8 28.0 0.0 0.0 3.7 70.0 984.6 62.0 0.9 98 4 4 | + | |
- | 2685 2011-04-18 01:15 3C286 237.8 60.7 24.00 2.6 -1.9 0.0 0.0 27.0 -7.8 28.0 0.0 0.0 3.7 70.0 984.6 62.0 0.3 157 2 2 | + | |
- | 2686 2011-04-18 01:17 3C286 238.2 60.5 24.00 2.6 -1.9 0.0 0.0 27.0 -7.3 28.0 0.0 0.0 3.7 70.0 984.6 62.0 -0.2 208 4 4 | + | |
- | 2687 2011-04-18 01:25 W3MAIN 369.2 23.5 23.41 -0.0 -0.5 0.0 0.0 27.0 -7.3 28.0 0.0 0.0 3.7 70.0 984.1 62.0 0.0 158 2 2 | + | |
- | 2688 2011-04-18 01:35 3C345 131.9 75.4 23.41 -0.0 -0.5 0.0 0.0 27.0 -7.3 28.0 0.0 0.0 3.5 72.0 984.1 63.0 -0.0 164 4 4 | + | |
- | 2689 2011-04-18 01:38 3C345 133.5 75.7 23.41 1.2 7.7 0.0 0.0 27.0 -7.3 28.0 0.0 0.0 3.5 71.0 984.1 62.0 -0.0 191 2 2 | + | |
- | 2690 2011-04-18 01:39 3C345 134.2 75.8 23.41 1.2 7.7 0.0 0.0 27.0 -7.6 28.0 0.0 0.0 3.5 71.0 984.1 63.0 -0.0 238 4 4 | + | |
- | 2691 2011-04-18 01:42 3C345 136.1 76.2 25.06 0.4 6.1 0.0 0.0 27.0 -7.5 28.0 0.0 0.0 3.6 71.0 984.1 62.0 0.4 230 4 4 | + | |
- | 2692 2011-04-18 01:45 3C345 137.8 76.4 25.06 -0.1 7.3 0.0 0.0 27.0 -7.5 28.0 0.0 0.0 3.6 71.0 984.1 62.0 0.5 232 2 2 | + | |
- | .... | + | |
- | </code> | + | |
- | ===Logbook.py=== | + | Looking at single scan might be appropriate for checking the data during an observation, but for calibration or flux density monitoring a more automatic way is preferred. There is a collection of scripts and programs that can be used to perform all the tasks to obtain flux density calibrated data. The scripts are located in /opt/bin on the **Observer-PCs**. |
- | Logbook.py lists by default the last 1000 scans. This might take some time, because Logbook.py is reading the original MBFITS data files to get the information. Using | + | ==== Raw Data Processing ==== |
- | <code> | + | |
- | Logbook.py 100 | + | |
- | </code> | + | |
- | it lists only the last 100 scans. The time range can be restricted by the option | + | |
- | <code> | + | |
- | Logbook.py 10000 tstart=YYYY-MM-DD tstop=YYYY-MM-DD | + | |
- | </code> | + | |
- | where "tstart" marks the more recent time (if not specified the print out starts today) and "tstop" specifies the stop time in the past. The 10000 is just chosen to be big enough to cover most time ranges. Logbook.py can be used only to produce logs from more recent data because it searches only in /daten/Raw. | + | |
- | For example: | + | The scripts and programs mostly use a file that contains all the scan numbers to be reduced. The scripts **dblog2scan.py** or **dblog2scan_wea.py** produce such lists from observing logs written Obslogger. |
- | <code> | + | |
- | obs2@observer3:~$ Logbook.py tstart=2011-04-18 tstop=2011-04-18 | + | In dblog2scan.py one can optionally restrict the frequency by the receiver version and frequency in GHz, if the log contains entries at different frequencies. |
- | # SCAN SUB OBJECT SCANTYPE FEBE PROJECT DATE_OBS MJD | + | |
- | .... | + | |
- | 2692 2 3C345 FOCUS P13mm-PBE 46-09 2011-04-18T01:45:17 55669.0731 | + | |
- | 2691 4 3C345 POINT P13mm-PBE 46-09 2011-04-18T01:42:56 55669.0715 | + | |
- | 2690 4 3C345 POINT P13mm-PBE 46-09 2011-04-18T01:40:05 55669.0695 | + | |
- | 2689 2 3C345 FOCUS P13mm-PBE 46-09 2011-04-18T01:38:44 55669.0686 | + | |
- | 2688 4 3C345 POINT P13mm-PBE 46-09 2011-04-18T01:36:22 55669.0669 | + | |
- | 2687 2 W3MAIN ONOFF P13mm-FFTS 46-09 2011-04-18T01:26:08 55669.0598 | + | |
- | 2686 4 3C286 POINT P13mm-PBE 46-09 2011-04-18T01:17:28 55669.0538 | + | |
- | 2685 2 3C286 FOCUS P13mm-PBE 46-09 2011-04-18T01:16:08 55669.0529 | + | |
- | 2684 4 3C286 POINT P13mm-PBE 46-09 2011-04-18T01:13:45 55669.0512 | + | |
- | .... | + | |
- | </code> | + | |
- | ===log.py=== | + | |
- | log.py is a small script that prints some more detailed information than Logbook.py. | ||
<code> | <code> | ||
- | Usage: log.py scan1 scan2 [fdir=<PATH>] | + | ubach@observer4:~$ dblog2scan_wea.py |
- | Options: | + | Usage: dblog2scan.py log-file [receiver] [freq GHz] |
- | fdir=<PATH>: if the data are not in /daten/Raw any more | + | |
- | </code> | + | |
- | + | ||
- | It can be also used to produce a log from older data in the archival directories, /datan/Raw/Raw-YYYY-MM specifying "fdir". | + | |
- | + | ||
- | For example: | + | |
- | <code> | + | |
- | obs2@observer3:~$ log.py 2684 2692 | + | |
- | # *** Date: 2011-04-18 Project-ID: 46-09 Observer-Operator: ALEX *** | + | |
- | # ** Temp = 3.7 deg C Humid = 70.0 % Press = 984.6 hPa | + | |
- | #Scan Sub Source UT LST FE-BE Mode Freq Type Sw-mode AZI ELV Col* NULE z-Lin | + | |
- | 2684 4 3C286 01:13:45 15:23:59 P13mm-PBE CONTINUUM 24000.00 POINT TOTP 236.9 61.0 4.3 2.1 -7.8 | + | |
- | 2685 2 3C286 01:16:08 15:26:36 P13mm-PBE CONTINUUM 24000.00 FOCUS TOTP 237.8 60.7 2.6 -1.9 -7.8 | + | |
- | 2686 4 3C286 01:17:28 15:27:44 P13mm-PBE CONTINUUM 24000.00 POINT TOTP 238.2 60.5 2.6 -1.9 -7.3 | + | |
- | 2687 2 W3MAIN 01:26:08 15:36:25 P13mm-FFTS SPECTROSCO 23407.18 ONOFF TOTP 369.2 23.5 -0.0 -0.5 -7.3 | + | |
- | 2688 4 3C345 01:36:22 15:46:40 P13mm-PBE CONTINUUM 23407.18 POINT TOTP 131.9 75.4 -0.0 -0.5 -7.3 | + | |
- | 2689 2 3C345 01:38:44 15:49:18 P13mm-PBE CONTINUUM 23407.18 FOCUS TOTP 133.4 75.7 1.2 7.7 -7.3 | + | |
- | 2690 4 3C345 01:40:05 15:50:24 P13mm-PBE CONTINUUM 23407.18 POINT TOTP 134.2 75.8 1.2 7.7 -7.6 | + | |
- | 2691 4 3C345 01:42:56 15:53:15 P13mm-PBE CONTINUUM 25056.25 POINT TOTP 136.0 76.1 0.4 6.1 -7.5 | + | |
- | 2692 2 3C345 01:45:17 15:55:56 P13mm-PBE CONTINUUM 25056.25 FOCUS TOTP 137.7 76.4 -0.1 7.3 -7.5 | + | |
+ | Give a frontend designation (e.g. 28.1) and additional a frequency | ||
</code> | </code> | ||
- | ===== Reducing a number of scans at once ===== | + | **dblog2scan.py** will just produce the files scans (input file for weather.py and corr_point.py) and scanlist (to be attached to the reduce.par file), **dblog2scan_wea.py** will produce an additional weather.dat file with weather information for each scan. The weather file is useful for the opacity correction described below. |
- | + | ||
- | Looking at single scan might be appropriate for checking the data during an observation, but for calibration or flux density monitoring a more automatic way is preferred. There is a collection of scripts and programs that can be used to perform all the tasks to obtain flux density calibrated data. The scripts are located in /home/obs2/bin on the observer3-PC. | + | |
- | + | ||
- | ==== Raw Data Processing ==== | + | |
- | + | ||
- | The scripts and programs mostly use a file that contains all the scan numbers to be reduced. The scripts **log2scan.py** or **logbook2scan.py** produce such lists from observing logs written by log.py or Logbook.py, respectively. | + | |
- | + | ||
- | In log2scan.py one can optionally restrict the frequency if the log contains entries at different frequencies. | + | |
- | <code> | + | |
- | log2scan.py | + | |
- | Usage: log2scan.py log-file [freq GHz] | + | |
- | </code> | + | |
---- | ---- | ||
+ | \\ | ||
The script **reduce.py** can be used to reduce a number of given scans using the toolbox with a list of options. Some example parameter files are stored in /home/obs2/flux_monit/reduce-par. E.g., reduc28.par for the 2.8cm SFK receiver. Calling just reduce.py prints out some help as well. | The script **reduce.py** can be used to reduce a number of given scans using the toolbox with a list of options. Some example parameter files are stored in /home/obs2/flux_monit/reduce-par. E.g., reduc28.par for the 2.8cm SFK receiver. Calling just reduce.py prints out some help as well. | ||
<code> | <code> | ||
- | obs2@observer3:~$ reduce.py | + | obs2@observer9:~$ reduce.py |
Usage: reduce.py par-file | Usage: reduce.py par-file | ||
Line 258: | Line 196: | ||
Since the amplitudes from the toolbox are in units of the Tcal one just have to multiply the given numbers by the Tcal to get the correct system temperature. The reduce.py procedure provides an all.fit file that contains all the single fit-files. The further programs work with this all.fit. | Since the amplitudes from the toolbox are in units of the Tcal one just have to multiply the given numbers by the Tcal to get the correct system temperature. The reduce.py procedure provides an all.fit file that contains all the single fit-files. The further programs work with this all.fit. | ||
- | The script **weather.py** reads the weather information for the MBFITS files, multiplies the Tsys found in the all.fit by a given Tcal and prints out a weather.dat and a LIST.tsys, with the opacity information for each scan that is listed in the scanlist-file. The minimum zenith opacity is fitted from the Tsys vs. airmass distribution and is presented in the plot Opacities.eps. Weather.eps contains the weather info. | + | The script **weather.py** reads the weather information from the weather.dat file or directly from the MBFITS files. It multiplies the Tsys found in the all.fit by a given Tcal and prints out a LIST.tsys, with the opacity information for each scan that is listed in the scanlist-file. When the weather data is read from the raw MBFITS file the script will write a new weather.dat for alter use. The minimum zenith opacity is fitted from the Tsys vs. airmass distribution and is presented in the plot Opacities.eps. Weather.eps contains the weather info. |
<code> | <code> | ||
- | obs2@observer3:~$ weather.py | + | obs2@observer9:~$ weather.py |
Task to read weather data from MBFITS files, | Task to read weather data from MBFITS files, | ||
save them in ASCII format, and compute LIST.tsys. | save them in ASCII format, and compute LIST.tsys. | ||
Line 289: | Line 227: | ||
<code> | <code> | ||
- | obs2@observer3:~$ corr_point.py | + | obs2@observer9:~$ corr_point.py |
Task to reduce fit-files from Peters Toolbox | Task to reduce fit-files from Peters Toolbox | ||
Line 307: | Line 245: | ||
</code> | </code> | ||
- | For example the Tcal at 2.8cm is 7.5 K and scan numbers are stored in scanlist | + | For example the Tcal at 2.8cm is 7.5 K and scan numbers are stored in the file scans. |
<code> | <code> | ||
- | obs2@observer3:~/ubach/measurements/2011_03_16_poi/2.8cm/0.Raw$ corr_point.py scanlist 7.5 | + | obs2@observer3:~/ubach/measurements/2011_03_16_poi/2.8cm/0.Raw$ corr_point.py scans 7.5 |
**************************************************** | **************************************************** |