Subject: Announce: Datascope: a Unix database system based on flat files
From: [email-masked] (Daniel Quinlan)
Date: 3 Jan 1996 12:47:21 GMT
Newsgroups: comp.lang.perl.announce,comp.lang.perl.misc

Datascope: A Relational Database System for Scientists 

Datascope is a database system with tcl and perl interfaces (hence
the cross-posts).  If the information below interests you, take a
look at 

	http://jspc-www.colorado.edu/software/software.html#datascope

for more information.

What it is: 

    Datascope is a relational database system in which tables are
    represented by fixed-format files.  Typically these files are plain
    ascii files; the fields are separated by spaces and each line is a
    record.  The format of the files making up a database is specified in
    a separate schema file.  The system includes simple ways of doing the
    standard operations on relational database tables:  subsets, joins,
    and sorts.  The keys in tables may be simple or compound.  Views are
    easily generated.  Indexes are generated automatically to perform
    joins, but may also be created precomputed and explicitly saved. 
    Fairly general expressions may be evaluated, and can be used as the
    basis of sorts, joins, and subsets,

    The system provides a variety of ways to use the data.  There are c,
    fortran, perl, and tcl interfaces to the database routines.  There are
    command-line utilities which provide most of the facilities available
    through the programming libraries.  There are a few GUI tools for
    editing and exploring a database.  And, since the data is typically
    plain ascii, it's also possible to just use standard UNIX tools like
    sed, awk, and vi.  

What it is not: 

    Datascope is not a full fledged multiple user database.  Access
    permissions are handled strictly through UNIX file permissions. 
    Multiple people may use a database at once, but only one user should
    be modifying any table at any time; the database does not enforce this
    limitation, however, and this can lead to the same kind of problems
    that you experience when two people edit the same file at the same
    time.  There are no transactions, no rollback, and no logging.  Update
    operations happen immediately (that is, as quickly as UNIX writes them
    to disk -- there is no attempt to deal with synchronization issues). 
    The system is intended for either fairly static databases, or
    databases where only one person is making updates. 

    Datascope is not SQL based.  It might be fairly simple to add an SQL
    interface using the basic routines, but Datascope provides the
    functionality of SQL in a different fashion. 

    Datascope is intended for relatively small databases.  It uses memory
    fairly prodigously, since it memory maps all tables which are
    referenced, and also keeps the indexes in memory.  This has not been a
    serious limitation so far, and as prices of memory and storage
    continue to fall, it shouldn't be in the future. 

Novel Features

Despite its limitations, Datascope has some novel features: 

       It is fairly small, conceptually simple, and fast.  The
       Datascope specific source code is around 12,000 lines of c
       (various support libraries bring this total a bit higher). 

       Interfaces to several languages (c, fortran, tcl and perl), a
       command line interface, and a GUI interface are part of the
       package. 

       It is freely available. 

       The keys to tables may include ranges, for instance, a
       beginning time and an ending time.  This is useful, perhaps
       essential, for time dependent parameters, like instrument
       settings.  Indexes may be formed on these ranges, and these
       indexes can considerably speed join operations.  (When two
       tables are joined by time range keys, the join condition is
       that the time ranges overlap.)

       The schema file, in addition to specifying the fields which
       make up tables, and the format of individual records in every
       table, may provide a great deal of additional information,
       including : 

	   short and long descriptions of every attribute and relation

	   null values for each attribute

	   a legal range for each attribute

	   units for an attribute

	   Primary and alternate keys for each relation. 

	   Foreign keys in each relation

    This additional information is useful for documenting a database and
    when exploring a new database.  It often makes it possible to form the
    natural joins between tables without explicitly specifying the join
    conditions. 

What is it good for? 

    Relational database systems are clearly a useful way of representing
    information, much more powerful than the traditional scientific
    approach of grabbag datafiles, log files, handwritten notes, and
    bizarre and idiosyncratic data formats.  However, most relational
    database systems are expensive, difficult to understand, difficult to
    administer, and subject to mysterious failures.  In addition, they
    require large amounts of disk space and preferably a dedicated
    machine.  These characteristics put them out of the ballpark for most
    casual users and most scientists with smallish budgets. 

    Datascope provides a cheap, easy, fairly intuitive way of moving from
    the traditional plethora of formats to a better approach which
    organizes the data, documents it, and provides powerful tools for
    manipulating it.  It should be useful to any one who needs to organize
    data, but can't afford the time, money or other resources which a
    commercial database system requires. 
-- 
Daniel Quinlan               		[email-masked]
University of Colorado, Physics Department/JSPC, Campus Box 583       
Boulder, CO  80309-0390			            303/492-4878