DisPerSE - persistent structures identification - Field data

vtk field format

2430-01-06T06:56:00+01:00

VTK formats are developed for the Visualization Tool Kit library (VTK) and can be used for 3D visualization with software such as VisIt or ParaView. Only regular grids field types can be converted to VTK (for particle type fields, convert to NDnet unstructured network format first and then use netconv -to vtk). Regular grids are stored as VTK image data and can be output in fours different VTK formats:

-vtk: the legacy format
-vtk_ascii: ASCII version of the vtk format
-vti: a more recently developed XML version of the vtk format,
-vti_ascii: ASCII version of the vtu format

The specifications for these formats can be found in this PDF file. See also here and there for additional information.

SDL-image format

2430-01-05T06:56:00+01:00

This input format is used to read 2D regular grids encoded in a popular picture format readable by SDL-image library (jpg, png, gif, ...). When using this format as input to mse, the field whose topology is computed is the luminosity L of the image, a weighted average of the RED, GREEN and BLUE components of the image:

L = 0.2989*RED + 0.5870*GREEN + 0.1140*BLUE

survey_ascii format

2430-01-04T06:56:00+01:00

This is a simple ASCII format whose main purpose is to easily read coordinates of discretely sampled astrophysical surveys (e.g. such as an SDSS galaxy catalog). It should be considered when using spherical coordinates systems or if distance is measured by redshift for instance.

In this format, particles properties are encoded in an ASCII array where each row corresponds to one particle and each column to one property. Each column must have a name defined in the header (the first line of the file, starting with character #). An survey_ascii file may look like this:

# ra dec z my_field
+2.115401e+02	+6.313433e-01	+2.666800e-01  1	
+2.115633e+02	+7.550188e-01	+1.259900e-01  0	
+2.115687e+02	+8.108763e-01	+3.646600e-01  0	
+2.117158e+02	+6.393598e-01	+1.143600e-01  1	
+2.116826e+02	+6.528485e-01	+2.455700e-01  1	
+2.116993e+02	+6.509297e-01	+1.199000e-01  0	
+2.115738e+02	+7.772653e-01	+3.240600e-01  0	
+2.116198e+02	+6.950604e-01	+1.987300e-01  0	
+2.116773e+02	+7.085776e-01	+2.561900e-01  1

The name of a column defines its role if it matches one of the following keywords:

header keywords
Keyword	Meaning
px	X coordinate of the particle
py	Y coordinate of the particle
pz	Z coordinate of the particle
vx	X component of the velocity
vy	Y component of the velocity
vz	Z component of the velocity
id	an index associated to the particle
ra	The right ascension of the particle
dec	The declination of the particle
z	The redshift of the particle.

When particles coordinates are defined with ra, dec or z (redshift), DisPerSE automatically transform them into cartesian coordinates using the standard LCDM model to compute distances (Omega_m=0.27, Omega_L=0.73 ). This transformation can be inverted on the output skeletons and networks using options -toRaDecZ and -toRaDecDist of skelconv and netconv.

FITS format

2430-01-03T06:56:00+01:00

This field format is used to read 2D or 3D regular grids of arbitrary dimensions in the popular Flexible Image Transport System (FITS) image format. A popular library for reading and writing FITS image in C or FORTRAN is CFITSIO, which must be installed for DisPerSE to be able to use this kind of files.

This file format is also used for storing HEALPIX tesselations of the sphere. DisPerSE will automatically detect if the FITS file contains a HEALPIX tesselation.

NDfield format

2430-01-02T06:56:00+01:00

This is the native binary format of DisPerSE. Functions for reading and writing NDfield format in C can be found within the file ${DISPERSE_SRC}/src/C/NDfield.c (see functions Load_NDfield and Save_NDfield).

When using the C functions from Disperse, data is loaded into the following C structure which is close to the actual structure of the file (see file ${DISPERSE_SRC}/src/C/NDfield.h):

#define ND_CHAR   (1<<0)
#define ND_UCHAR  (1<<1)
#define ND_SHORT  (1<<2)
#define ND_USHORT (1<<3)
#define ND_INT    (1<<4)
#define ND_UINT   (1<<5)
#define ND_LONG   (1<<6)
#define ND_ULONG  (1<<7)
#define ND_FLOAT  (1<<8)
#define ND_DOUBLE (1<<9)

typedef struct NDfield_str
{
 char comment[80];  // a comment on the data
 int dims[NDFIELD_MAX_DIMS];  // dimensions of the grid, must be [ndims,nparticles] when data represents sample particles coordinates (i.e. when fdims_index!=0)
 int ndims;  // number of dimensions of the space
 int n_dims;  // number of meaningfull values in dims array
 int fdims_index; // if 0, the field is a regular grid of dimensions dims, else the file contains the dims[0] coordinates of dims[1] particles.
 int datatype;  // type of the data (one of the ND_... defined above)
 double x0[NDFIELD_MAX_DIMS];  // origin of the bounding box
 double delta[NDFIELD_MAX_DIMS];  // extent of the bounding box
 char dummy[160];  // dummy data, for future extensions or for storing anything you want.

 void *val;  // pointer to data

 long nval;  // total number of particles (fdims_index==1) or pixels (fdims_index==0)
 int datasize;  // size in bytes of datatype type.
} NDfield;

The NDfield binary format is organized as follows (blocks are delimited by dummy variables indicating the size of the blocks for FORTRAN compatibility, but they are ignored in C):

NDnet binary format
field	type	size	comment
dummy	int(4B)	1	for FORTRAN compatibility
tag	char(1B)	16	identifies the file type. Value : "NDFIELD"
dummy	int(4B)	1
dummy	int(4B)	1
ndims	int(4B)	1	number of dimensions of the embedding space
dims	int(4B)	20	size of the grid in pixels along each dimension, or [ndims,nparticles] if data represents particle coordinates (i.e. fdims_index=1)
fdims_index	int(4B)	1	0 if data represents a regular grid, 1 if it represents coordinates of tracer particles
datatype	int(4B)	1	type of data stored (see below)
x0	double(8B)	20	origin of bounding box (first ndims val. are meaningfull)
delta	double(8B)	20	size of bounding box (first ndims val. are meaningfull)
dummy_ext	char(1B)	160	dummy data reserved for future extensions
dummy	int(4B)	1
dummy	int(4B)	1
data	size of datatype	N	data itself (N may be the number of pixels or ndism times the number of particles)
dummy	int(4B)	1

Possible data types
name	size (Bytes)	type	value
ND_CHAR	1	integer	1 (=1<<0)
ND_UCHAR	1	integer	2 (=1<<1)
ND_SHORT	2	integer	4 (=1<<2)
ND_USHORT	2	integer	8 (=1<<3)
ND_INT	4	integer	16 (=1<<4)
ND_UINT	4	integer	32 (=1<<5)
ND_LONG	8	integer	64 (=1<<6)
ND_ULONG	8	integer	128 (=1<<7)
ND_FLOAT	4	float	256 (=1<<8)
ND_DOUBLE	8	float	512 (=1<<9)

NDfield_ascii format

2430-01-02T06:56:00+01:00

A simple ASCII version of the NDfield format used to represent either uniform grids or particles coordinates. Data type is always interpreted as double precision floating point.

NDfield_ascii format

ANDFIELD COORDS	header (COORDS keyword is optional, if present, values are interpreted as coordinates, or else, as pixel values)
[N_1 ... N_ndims]	size of the grid along each of the ndims dimensions (in pixels) or number of dimensions and number of particles if COORDS keyword is in the header (= [ndims,nparticles])
BBOX [X0_1 ... X0_ndims] [delta_1 ... delta_ndims]	OPTIONAL: bounding box definition (origin and extent)
V_1 V_2	values (any number of values can be on the same line)
V_3	no particular formating is required
...	Repeat enough time to define a number of values equal to the number of pixels or the number of particles (if COORDS keyword is in the header)

Field files

2430-01-01T06:56:00+01:00

This file type is designed to store scalar fields sampled over regular grids (i.e. regular images in 2D, 3D and more) or, by extension, coordinates of tracer particles sampling an underlying density field. The main field format is NDfield, which allows to store any type of data (integer, simple or double precision floating point, ...) over arbitrary sized regular grids or as coordinates of tracer particles.

Available formats:

-NDfield (Read / Write):
This is the format used internally in DisPerSE, it is efficient and generic as it can store regular grids or sample particles coordinates indifferently.

-NDfield_ascii (Read / Write):
A simple ASCII version of the NDfield format, it is as versatile but data is always considered to be double precision floating point.

-FITS (Read only):
Used to read 2D or 3D regular grids of arbitrary dimensions in the popular Flexible Image Transport System (FITS) image format. Also used for HEALPIX tessellations of the sphere.

-survey_ascii(Read only):
A simple ASCII format whose main purpose is to easily read coordinates of discretely sampled astrophysical surveys (e.g. such as an SDSS galaxy catalog). Try this if you use spherical coordinates or if distance is measured by redshift for instance.

-SDL-image (Read only):
This input format is used to read 2D regular grids encoded in a popular picture format readable by SDL-image library (jpg, png, gif, ...).

-vtk, vtk_ascii, vti and vti_ascii (Write only):
These formats are binary and ASCII legacy and XML VTK formats that are readable by several 3D visualization tools, such as VisIt or ParaView for instance.

-NDnet (Write only):
Using this format, field files representing tracer particles coordinates can be converted to unstructured networks (use netconv to convert NDnet files to other network formats). Regular grids conversion is not implemented yet due to the huge size of the output network.