Rawtran

Rawtran is a wrapper around dcraw developed to produce images in the standard astronomical FITS format by processing of raw (CR2, CRW, MRW, etc.) photos.

Introduction

Selected digital cameras offers possibility to save raw photos, eg. only slightly pre-processed photos intended for further image processing. Rawtran is developed as a wrapper around dcraw due D.Coffin to produce FITS files from the raw photos. FITS files are widely used by astronomers and can be processed by many standard astronomical software by analogical way as ones acquired by an ordinary scientific CCD camera. FITS files produced by Rawtran are suitable for all photometry, spectroscopy or astrometry measurements.

Overview

Proper conversion of data acquired by common digital cameras has two key features:

Metadata or a description of physical parameters of frame like date and time, used optics etc. as the integral part of frames. It is absolutely necessary for any regular scientific processing. Therefore Rawtran carefully implements conversion of EXIF data enclosed to every raw photo and interprets its in scope of usual astronomical conventions.

Colours and its proper handling is important for both photometry or spectroscopy interpretations. Therefore Rawtran implements conversion from CIE 1931 XYZ colour space to common astronomical photometry bands (conversion from instrumental to XYZ is provided by dcraw). Saving frames in Colour FITS format is just side effect of the conversions.

Rawtran also implements a few convenience functions common in astronomical photometry for using of dark frames.

Conversion to Colour FITS

The output image is Colour FITS image. Because, there is no widely accepted convention or definition of the colour representation in FITS world, Rawtran uses Munipack's specification of Colour FITS:

The purpose of the conversion type is a simple manipulation (and representation) of colour images in FITS format. Colour FITS can be displayed by xmunipack (GUI for Munipack) or converted to PNG by fitspng (used to generate all images on this page).

Instrumental (Bayer's) data

The output image contains one 3D primary array without any photometric transformations. Four original bands are stored in the array with the structure: width ⨉ height ⨉ 4. The bands are storing Bayer's colours in the order: B (blue), G1 (green1), G2 (green2), R (red). According to astronomical terminology, the single bands are in an instrumental photometric system.

The conversion type provides absolutely raw access to data in FITS format. One offers a direct conversion between file formats (it rearranges bytes only) and does NOT modify data itself. To be useful, transform the data to any known colour space by using of a transformation matrix (included in the source code of dcraw.c) or derive the transformation matrix yourself.

Conversion and selection of a single band

The conversion is intended for representation of the intensity in the selected colour band. The output image is a grey-scale and data are stored as two dimensional FITS primary array.

Note, that all data are slightly modified as result of a colour interpolation on Bayer's mask. Use -X "-h" to switch-off the interpolation.

All data must be supposed as RELATIVE quantities only.

A required photometry band can be selected with -c switch.

Instrumental bands (Ri,Gi1,Gi2,Gi,Bi)

Selected colours band is simply copied out from the the Bayer's mask. Gi is derived as the arithmetical mean Gi = (Gi1 + Gi2)/2. These instrumental filters are specific for an unique camera model.

CIE XYZ 1978 colour bands

Instrumental bands are difficult to reconstruct a standard colours or photometric information. Therefore there is possibility to get a standard band. Colours are converted to primary col or space XYZ.

Y component represents luminance. Its spectral profile is similar to humans eye spectral sensitivity under well-light conditions (photopic vision).

Johnson photometric system (B,V,R)

With exact definition of XYZ spectral profiles, we can derive of colour transformation matrix to another colour-space. Rawtran has built-in a transformation matrix from XYZ to Johnson UBVRI system (wide-band astronomical photometry system).

Derived bands (unfiltered, scotopic)

It is also possible to specify of a wide-spectral band (like the human eye or an unfiltered CCD image) or to simulate scotopic vision (like human eye under low-light conditions). The visual difference between colour images (or Y component) and the scotopic one illustrates Purkyňův effect.

Command line options

-c [X|Y|Z|R|V|B|u|s] (standard bands)
-c [Ri|Gi|Gi1|Gi2|Bi] (instrumental bands)
Selects a colour band:
X,Y or Z
CIE XYZ 1931 colour-space. Y is the human eye sensitivity under well-light conditions.
B,V or R
Johnson astronomical photometry system
u
Unfiltered, integrated over full camera spectral sensitivity, u = X + Y + Z for all pixels. Recommended to use of -b 32.
s
Scotopic. The human eye sensitivity under low-light conditions, s = 0.36169 Z + 1.18214 Y - 0.80498 X for all pixels.
Ri,Gi,Gi1,Gi2 or Bi
Instrumental, camera specific colour in Bayer's mask. The mean green is Gi=(Gi1+Gi2)/2 for all pixels.
-B [8|16|32|-32]
Select bits per pixel of an output image. Possible values are: 8 (0-255), 16 (0-65535), 32 (0-4294967296) and -32 (-10^38 .. 10^38), (values in braces indicates numerical range). An optimal bitpix for raw (instrumental) data is 16 (default) which covers a full range of digital cameras. Some out of range (rare) values will be suppressed (cut-off). The representation occupies width*height*2 bytes. The representing by float numbers (eg. -32) is recommended value for images intended for further processing because saves numerical precision and their numerical range (but occupies of twice more space with respect to 16). 8 bits reduces range (eg. suppress dynamical range) and 32 wastes a lot of storage place only.
-o file
Output file name. If the option is not used, the output is stored as rawtran.fits. Use dash '-' for redirection to the standard output. Precede the filename with exclamation point (!) to overwrite an existing file. Usually, to prevent shell expansion, it will be require to escape it: \!rawtran.fits.
-D RAW dark file
Use the raw photo as a dark frame. Rawtran will try to convert the file to PGM format by the recommended command dcraw -D -4 -j -t 0 file.RAW and pass it to dcraw -K file.pgm. Note that sometimes can be necessary to switch-off automatic frame rotation using -A "-t 0" option. See Dark Frame section for more info.
-E FITS dark file
Use the FITS file as a dark frame. The FITS file must be previously converted as dcraw -D -4 -j -t 0 dark.RAW. See Dark Frame section for more info.
-C options
Conversions options for dcraw. Default is '-4 -o 5' for standard photometry filters and '-4 -D' for instrumental bands. Defaults are usually satisfactory.
-X options
Specify options to conversion by dcraw itself. One represents default parameters: '-q 3 -w'. See also -A.
-A options
Specify additional options to dcraw itself. If you need pass more options, enclose ones to quotes or apostrophes. In doubts, use -A "-v" to show the detailed trace of the conversion provided by dcraw.
-h, --help, --version
Show summary of options or current versions.

Splitting of conversion parameters for dcraw on three groups of options -C,-X,-A is conventional. Main purpose is separating important options -C,-X and optional -A to prevent potential mistakes. Changing of the important parameters is recommended only for experienced users.

Return value

The zero indicates successful conversion. A non-zero value is returned when an error is occurred during conversion. If the utility is launched without options or with -h switch, zero means that Rawtran's internal checker can run dcraw binary and non-zero value otherwise.

Examples of usage

It converts a raw photo to colour FITS. The results is displayed by xmunipack:

$ rawtran -o IMG_0666.fits IMG_0666.CR2
$ xmunipack IMG_0666.fits

It converts an image to grey-scale meaning to be an equivalent seen by human eye at good light conditions:

$ rawtran -c Y -o IMG_0666.fits IMG_0666.CR2

It converts an image to be equivalent of Johnson's B filter:

$ rawtran -c B -o IMG_0666.fits IMG_0666.CR2

It extracts the (mean) green component (camera specific) of the raw photo:

$ rawtran -c Gi -o IMG_0666.fits IMG_0666.CR2

The output file can be overwritten:

$ rawtran -o '!IMG_0666.fits' IMG_0666.CR2

A large set of raw photos can be converted with help of shell functionality:

$ for A in *.CR2; do rawtran -o ${A%CR2}fits ${A}; done

Gallery

All this gallery examples has been generated by processing of the reference raw photo IMG_5952.CR2:

$ rawtran IMG_5952.CR2 -o IMG_5952.fits
$ rawtran IMG_5952.CR2 -o IMG_5952_X.fits -c X
$ rawtran IMG_5952.CR2 -o IMG_5952_Y.fits -c Y
$ rawtran IMG_5952.CR2 -o IMG_5952_Z.fits -c Z
$ rawtran IMG_5952.CR2 -o IMG_5952_B.fits -c B
$ rawtran IMG_5952.CR2 -o IMG_5952_V.fits -c V
$ rawtran IMG_5952.CR2 -o IMG_5952_R.fits -c R
$ rawtran IMG_5952.CR2 -o IMG_5952_Bi.fits -c Bi
$ rawtran IMG_5952.CR2 -o IMG_5952_Gi.fits -c Vi
$ rawtran IMG_5952.CR2 -o IMG_5952_Ri.fits -c Ri
$ rawtran IMG_5952.CR2 -o IMG_5952_s.fits -c s
$ rawtran IMG_5952.CR2 -o IMG_5952_u.fits -c u

$ fitspng IMG_5952.fits -fl 0,20000 -s 10 -o IMG_5952.png
$ fitspng IMG_5952_X.fits -fl 0,20000 -s 10 -o IMG_5952_X.png
$ fitspng IMG_5952_Y.fits -fl 0,20000 -s 10 -o IMG_5952_Y.png
$ fitspng IMG_5952_Z.fits -fl 0,20000 -s 10 -o IMG_5952_Z.png
$ fitspng IMG_5952_R.fits -fl 0,20000 -s 10 -o IMG_5952_R.png
$ fitspng IMG_5952_V.fits -fl 0,20000 -s 10 -o IMG_5952_V.png
$ fitspng IMG_5952_B.fits -fl 0,20000 -s 10 -o IMG_5952_B.png
$ fitspng IMG_5952_s.fits -fl 0,20000 -s 10 -o IMG_5952_s.png
$ fitspng IMG_5952_u.fits  -fl 0,40000 -s 10 -o IMG_5952_u.png
$ fitspng IMG_5952_Ri.fits -fl 128,1000 -s 10 -o IMG_5952_Ri.png
$ fitspng IMG_5952_Gi.fits -fl 128,1000 -s 10 -o IMG_5952_Gi.png
$ fitspng IMG_5952_Bi.fits -fl 128,1000 -s 10 -o IMG_5952_Bi.png
IMG_5952

Colour FITS

IMG_5952_X

X component

IMG_5952_Y

Y component

IMG_5952_Z

Z component

IMG_5952_R

R filter

IMG_5952_V

V filter

IMG_5952_B

B filter

IMG_5952_Ri

R Bayer instrumental

IMG_5952_Gi

G Bayer instrumental

IMG_5952_Bi

B Bayer instrumental

IMG_5952_u

Unfiltered

IMG_5952_s

Night vision

Dark frame

Rawtran provides convenience functions for handling of dark frames.

The basic way is use of -D option which launch dcraw to create PGM file representing of the dark. While dcraw sometimes saves results to PPM file, a transparent conversion to PGM can be required. Therefore the time duration of the procedure will depend on the conversion and generally one will be twice slower than passing a PGM file directly by -A "-K file.pgm". An user is responsible to provide correct dark frame.

A little bit advanced way is to use a FITS file as dark frame which is on the fly converted to PGN. Purpose of the option is to offer possibility for averaging of darks. A hypothetical session can be:

$ rawtran -c P -A "-j -t 0" -o dark1.fits dark1.CR2
$ rawtran -c P -A "-j -t 0" -o dark2.fits dark2.CR2
$ munipack dark -o dark.fits dark?.fits   # or equivalent
$ rawtran -E dark.fits -o light.fits light.CR2

First two lines (or possible more) converts dark frames to FITS format using of -c P option. Than the created FITSes are averaged (munipack utility is only an example, any another software can be used). The averaged file is used as the dark frame.

Download and installation

The tar-ball or the development repository with a source code is freely available under GPL2 license. cfitsio library with files required for development (headers, static libraries) is necessary for building. Building itself does not depends on dcraw.

A recommended way of local installation from the tar-ball is:

$ tar zxf rawtran-X.Y.Z.tar.gz
$ cd rawtran-X.Y.Z/
$ autoreconf -i   # for Mercurial
$ ./configure CFLAGS="-O4 -DNDEBUG"
$ make
# make install

The last step must be executed under root account. Both binary and man page are installed under /usr/local tree. It would be nice to keep the source directory for a case of later uninstalling.

Building under RPM-bases distributions (RHELL, Fedora) requires place of Rawtran tarball to a directory where RPM places sources (rpmbuild), unpack the spec file and we can use the following commands to build:

$ rpmbuild -bb rawtran.spec

where the package must by placed in your RPM building directory.

Rawtran runs under any Unix-like operating system (all flavors of GNU/Linux and BSD, Solaris, Mac OS X, etc). Usage under Windows or DOS has not been reported yet. Rawtran packages can be found in Debian and Ubuntu.

Implementation details

By design, Rawtran is a wrapper around dcraw. One internally launches dcraw as a sub-process and grabs its output. The info about image (EXIF information) is obtained as:

 dcraw -i -v -c

The image data oneself are imported by one from these alternatives:

 dcraw -4 -c -o 5    (standard bands)
 dcraw -4 -c -D      (instrumental bands)

Passed parameters can be changed with -C option (except -c).

Rawtran internally represents all image data as 4-byte real (with precision of 7 decimals and the maximum value 1033) during all numerical operations. Values stored in FITS files are rounded to a nearest integer (if necessary).

See also

Fitspng is a converter from FITS to PNG. Munipack is a general utility to work with FITS images.

Development notes can be found in Hroch's diary.

License

Rawtran is free software licensed under the GNU General Public License. This gives you the freedom to use and modify Rawtran to suit your needs.

Copyright © 2007 — 2018, F. Hroch, Institute of Theoretical Physics and Astrophysics, Masaryk University, Brno, Czech Republic.