SRE-Filter

The SRE-Filter For GoServe Web Server

Users guide



1) Introduction.

Welcome to SRE-Filter, version 1.2h (and counting). For those of you who've stumbled upon this document, :
SRE-FILTER is a HTTP (Web) Server for OS/2. Written in REXX, SRE-FILTER is designed to run under the IBM EWS GoServe program.
Narrowing our focus just a bit ...
SRE-FITLER is designed to be a relatively full-featured package for non-cutting edge, small-to-medium load sites being maintained by non-professionals
In fact, SRE-Filter has many obscure, and useful, features. These include: At this point your are either long gone, or still interested. Assuming the latter, the goal of this document is to outline some of the possibilities of SRE-Filter, and point you to the relevant documentation. Before we do that, it is de-rigeur to ask you to read the disclaimer.
Bored already? You may find the FAQ more informative, the manual more immediately useful, and the list of documentation files a useful index.
Not bored? You can proceed linearly through this documentation, or you can use the table of contents to jump to the desired section.

/ Are you looking for a quick start? /

Although this guide attempts to be succinct, there are a fair amount of details to bog down the unwary.
So... if you really don't need all those paranoid access controls, and your needs don't require refining, re-refining, and otherwise mangling requests, you probably can skim right through & and over this user guide! That is, if you are reasonably careful during installation, and answer all the questions that the install program asks, you'll be ready to go. I suppose you might want to read the installation section (section 3), but even that's not absolutely necessary (the INSTALL program will cover most of the details).

Let me add one proviso: if you want to use server side includes, you should carefully read the relevant section of the SRE-Filter manual. They can do a lot for you...






2) Table of Contents ------------

  1. Introduction
  2. Table of contents
  3. Installation
  4. Multiple Hosts
  5. Controlling Access
  6. Resolving requests
  7. Server side includes
  8. Server side processing
  9. Uploads, HEAD method requests, and other methods
  10. Pre and Post filter processing
  11. The synopsis of features
  12. The disclaimer and acknowlegments


...... ...

3) Installation

All good things must start somewhere, and that means you have to install SRE-Filter before you can use it (or have you already done that).

First, have you installed GoServe? I'll assume that you have.
Second do you have the most recent copy of SRE-Filter (if you have a slow connection, you might need to download SRE-Filter in pieces)? Assuming that you have, you should create (or clear out) a temporary directory, and UNZIP the (possibly several) SRE-Filter distribution files.

  1. Look at the READ.ME file that you just unzipped -- it outlines the installation steps (more impatient souls can skip this file with little ill effect).
  2. Ready to go? Run the INSTALL program from an OS/2 prompt (be sure you are in the temporary directory, or you might end up running somebody else's installation program!).
  3. INSTALL is self explanatory. It does require that you select the working directory and the data directory. These are explained in the GoServe documentation, or in the SRE-Filter manual.
  4. INSTALL also let's you set a few parameters, and will ask if you want to add a SUPERUSER entry to SRE-Filter's username database. We especially recommend adding the SUPERUSER entry, without it you may have a hard time accessing SRE-Filter's configurator!
When you are done installing, it's a good idea to review the installation notes contained in the INSTALL program.

Return to table of contents.



...... ...

4) Multiple Hosts

SRE-Filter is a multi-host (multi-homing) web server -- it can handle requests for several IP hosts simultaneously. This does require some configuration; in particular, you must define a host nickname for each host. Note that each host can be a unique IP address, or can be an IP alias.

The easiest way to add (and remove) hosts is through the Multiple Hosts section of the simple configurator. You may also want to examine the description of the HOSTS parameter in INITFILT.DOC, and the discussion of hosts in the SRE-Filter manual.

Return to table of contents.





...... ...

5) Controlling Access to your Site

SRE-Filter has several levels of access control, including Logon controls, URL-Specific access controls, and file specific access controls. In addition, you can identify resources that are open to the public.
Type of
control
Description Configuration Information?

LOGON controls At the coarsest level, you can require all clients to LOGON. In most cases, logging on requires that the client provide a username and password, which are then compared against the SRE-Filter user database. If there is no match, SRE-Filter will return a simple you are not authorized response. Alternatively, you can construct a customized response.

Since this logon requirement is a bit of a hassle, you can specify OWNERS and In-house clients; for whom logon requirements are waived. This specification is by IP address, with the possibility of using wildcards to grant access rights access to entire domains.

The Logon Controls section of the simple configurator can be used to turn enable this Logon requirement, to add and remove users, and to add and remove In-house clients.

Advanced User Notes: The following SRE-Filter parameters may be of interest (see INITFILT.DOC for details).

  • LOGON_LIMITS: limits the number of logon attempts per minute
  • DNS_CHECK: denies access to clients with no IP Name.
  • UNALLOWED_IPS: identify IP addresses that will not be allowed onto your site.
  • THE_REALM: the default REALM name (can be changed with the simple mode configurator)

  • Public areas The basic idea is that before logon or access_controls are checked, SRE-Filter checks if the requested URL (the request string) points to a PUBLIC area. If so, no logon or access controls are attempted.
    In a sense, the PUBLIC areas are purposely placed outside of the protection of SRE-Filter's various access controls.
    You can use the Public Areas section of the simple mode configurator to add and remove public areas. For a more nuanced control, you probably will want to use the Modify Initializaton Parameters section of the the intermediate configurator -- look for the sub-section on the PUBLIC_URLS parameters.

    Advanced users note: Significant performance advantages can be realized by using the SREFQUIK variant of SRE-Filter. This requires that you establish as many literal PUBLIC_URLS as possible. The intermediate mode configurator can be used to accomplish this.

    Since many sites will contain a continuum of private and public resources, this coarse level of control will often be inappropriate. SRE-Filter offers two finer methods of access controls: URL-specific access controls, and the HTACCESS method.

    URL-specific access controls URL-specific access controls are based on a comparison between the desired URL (the request string sent by the client) and a a list of URLs . This list (which may contain wildcarded entries) dictates who can access the various resources (files, etc.) on your site.

    Given a match, SRE-Filter will then compare the list of resource privileges assigned to the URL to the client privileges granted to the client. In general, the client must have at least one client privilege that's also in the resource privilege list (though you can specify more complicated conditions). If no such privilege exists, SRE-Filter returns an unauthorized access response. This response can be a generic response, a customized-for-your-site response, or a customized-for-this-URL response.

    The Access Controls section of the simple configurator lets you enable access controls, as well as choose between the generic and customized response. You can also use the simple mode to add and remove entries to the list of URL-specific access controls.

    You should use the Modify URL-Specific Access Controls section of the intermediate mode configurator to specify customized-for-this-URL responses, or if you want to specify more complicated conditions.
    Note that the list of URL-specific access controls is also used for several other purposes (such as granting URL-specific permissions); a task that the intermediate configurator is designed to do.

    Client privileges are granted when you set up user entries. You may also want to use the intermediate mode configurator to modify the INHOUSEIPS_PRIVS and PUBLIC_PRIVS parameters. If you want to get fancy, see ADDPRIVS.DOC for details on how you can specify dynamic client privileges.


    The HTACCESS method The HTACCESS method (which is something of a standard) uses special HTACCESS files located in the directory (and in the parent directories) of the requested file. This (or these) HTACCESS files contain information on who has access rights to files in the directory (and in child directories). HTACCESS files can be created using your favorite text-editor. You might want to read some documentation on the HTACCESS method of access control. Alternatively, you can use the intermediate configurator's Modify HTACCESS Contols options to change HTACCESS files.

    The choice between the URL-specific and HTACCESS methods is a matter of taste and convenience -- controlling directories is a sure way to prevent access to files, but controlling URL's is more flexible (if your physical directory structure changes more frequently then your pages). In fact, these two methods (URL-specific and directory-specific) can be used jointly, which implies multiple checks. But be careful when using both methods, it is possible to get stuck in authorization loops!

    Since URL-specific controls are native to SRE-Filter, they will tend to be faster then the HTACCESS method. Therefore, our recommendation is to use the URL-specific method.
    Return to table of contents.
    
    
    
    ...... ...

    6) Resolving requests

    Since all kinds of requests might show up on your server's doorstep, SRE-Filter offers a slew of techniques and tricks for resolving requests. These include specifying your home-page, indicating directory specific documents, establishing virtual directories, redirection to other servers, and revising the request string.
    The problem Solutions

    A document was not specified You can use the Default Documents section of the simple configurator to:
    • identify your home page -- it is sent in response to empty requests.
    • identify possible files (including directory specific file names) to use when a request for a directory appears
      .. or, generate a cachable directory listing instead (see DIR.DOC for
    • Advanced Users Note: see the description of the NOEXT_TYPE parameter (in INITFILT.DOC) for an alternative approach.

    Document could not be found By default, SRE-Filter will:
    • send a generic response that mentions the colloquial name of your site
    Alternatively, you can
    • modify the generic response, or
    • create a customized document not found response file.
    Use the Site Identification variables section of the simple configurator to modify the colloquial name, the generic response, and the document not found response file.

    A document has been moved to another location SRE-Filter uses aliases to redirect requests to another URL. This URL need not be on the same server, and it need not have the same name. You can use the Home Directory, virtual directories, and redirection section of the simple configurator to create these redirection aliases.

    You want to catch common mistaakes SRE-Filter uses aliases to specify local redirections: which involve textual substitutions in the recieved request string. Among other advantages, this gives you quite a bit of control over how "SRE-Filter facilities, and external procedures" percieve the request. The intermediate mode's Modify Redirection Aliases section can be used to create these local redirection aliases.

    You want to specify a ~ directory The home directory is used as a text replacement whenever a ~ is encountered in a request string. A typical value of this would be "Users/".

    Further refining the use of home directory, you may wish clients to have access to particular subdirectories of your "Users" directories. For example, all "students" may have space on the server machine, some of which is used for web, and some for "personal" purposes. The goal is to give clients direct access to the "web" related sub-directories but not to the "personal" sub-directories.

    To modify these home directory options, see the Home directory, etc. section of the simple configurator


    You want to transfer files from outside of GoServe's data directory By default, SRE-Filter will match a requested URL to a file in the GoServe data directory. While a good security feature (files not in or under the GoServe data directory are inaccessible), this can be an inconvenience.

    To remedy this inconvenience, one can define virtual directories

    Basically, SRE-FILTER will compare the starting portion of the client's request to see if it matches one of your virtual directory entries. If it does, the target directory listed in the matching entry is used (instead of the data directory). Thus, you can make available a wide, but controllable, set of directories (on or LAN accessible from) your server.

    To create virtual directories, see the Home directory, virtual directories, and redirection section of the simple configurator.

    Advanced users notes

  • Virtual directories can point to "remote" directories on other http servers -- SRE-Filter will attempt to retrieve the file from the remote server, without using a redirection (see the SRE-Filter manual for details).
  • Although somewhat obsolete, you can use special transfer alias to transfer selected files from outside the GoServe data directory (see the SRE-Filter manual for details).
  • Return to table of contents.

    
    
    
    ...... ...

    7)Server Side Includes

    SRE-Filter supports a variety of server side include (SSI) features.
    The feature Discussion More details

    Caching of documents that contain SSIs. To improve performance, SRE-Filter will "cache" HTML documents that have server side includes (SSI). This SSI-caching is of the document after the SSIs have been performed. In simple cases SRE-Filter can send this cached file, without having to repeat the actual process of SSI lookups, etc. Needless to say, this can greatly improve server performance. You can use the Server Side Includes section of the simple configurator to enable SSI-caching.

    For a detailed discussion of SSI-Caching, see SSICACHE.HTM.


    Suppressing Server Side Includes You can suppress server side includes for everyone, or for selected resources.

    The NO_INCLUDE parameter can be used to suppress all SSI's. A less drastic approach is to use NO_SSI permissions (in your URL-Specific access controls).

    See INITFILT.DOC for a discussion of NO_INCLUDE. To set permissions, see Access Controls section of the intermediate configurator.

    Limiting SSIs to a subset of your HTML documents SRE-Filter can be instructed to attempt server side includes (SSIs) only on a subset of HTML documents; such as those with .SHT or a .SHTML extension. In other words, HTML documents with .HTM or .HTML extension will not be checked for server side includes. This can speed up file transfer (especially when used in conjunction with SSI-caching), but does require more care when naming html files. See the Server Side Includes section of the simple configurator to enable this SSI on SHTML files only feature.

    Advanced users notes:

  • If you wish to use extensions other then .SHT or .SHTML, you can change the SSI_EXTENSIONS parameter (described in INITFILT.DOC).
  • You may also need to add text/html mime type entries to the MEDIATYP.RXX file (see the SRE-Filter manual for details).

  • Headers and Footers You can include headers and footers in all your HTML documents. They can contain Server Side Include keyphrases (see below). You can set both headers and footers in the Server Side Include section of the simple configurator (or see the SRE-Filter manual for further details).

    The NSCA HTTPD-style server side include syntax The NSCA HTTPD-style server side include syntax is fully supported by SRE-Filter. These include:
    • INCLUDEing files
    • Reporting size and creation date of files
    • Displaying selected system variables (such as current date and time, and the servername and client's IP addresses).
    • Including the output of CGI-Bin scripts
    The SRE-Filter manual also describes how to use NCSA HTTPD-style server side. You may also want to see TIMEFMT.DOC for a description of time and date display formats.

    The SRE-Filter syntax The SRE-Filter syntax overlaps the NCSA HTTPD-style syntax, but there are some important additions (note: you can use both types of SSIs in your documents). Additional features include:
    • INTERPRETing REXX code, either in-line or in external files
    • SELECTive excludes are supported
    • You can specify a large set of static variables to include in your documents
    • Several built-in text counters are available.
      Of particular interest, SRE-Filter can suppress augmenting the hit counter when:
    • Repetitive hits arrive from the same client
    • Requests arrive from OWNERS
    • OPTIONs passed to the html document can be incorporated into your document
    The SRE-Filter manual contains a detailed discussion of the SRE-Filter SSI syntax, including a discussion of the suppression of hits feature. The SRE-Filter FAQ contains a discussion of the various "hit" counting mechanisms available to SRE-Filter, as does COUNTER.DOC.

    You can use the Server Side Include section of the simple configurator to modify most of the static variables.

    Return to table of contents.

    
    
    
    ...... ...

    8)Server Side Processing

    SRE-Filter supports a variety of server side processing features. These include support for imagemaps and for searchable indices, CGI-Bin support, and support for SRE-Filter add-ons.
    The feature Discussion

    ImageMaps SRE-Filter supports NSCA-style imagemaps.

    The SRE-Filter manual contains a detailed discussion of how to specify and configure imagemaps. You can also play with SAMPMAP.HTM (a sample image map document, it uses the SAMPMAP.MAP imagemap file).


    Searchable Indices SRE-Filter is shipped with DOSEARCH, a moderately powerful text-file search engine. When used in conjunction with aliases, it is easy to implement searchable indices.

    The SRE-Filter manual contains a short discussion of how to setup searchable indices, as does the ALIASES.IN file.


    Special Requests SRE-Filter supports several special requests, all of which start with an exclamation point (!) The current set of special requests are:
  • !authorize !ping !statistics !ssi?option !host+ip.add.res.s !reset !save !variable?varname
  • See the SRE-Filter manual for details.

    CGI-Bin support SRE-Filter offers full support for the Common Gateway Interface.

    SRE-Filter offers a few enhancments, including:

  • an ability to specify alternative directories (within which to find CGI-Bin scripts)
  • an ability to use PERL scripts

  • SRE-Filter add-ons As an alternative to the somewhat clunky CGI-Bin interface, SRE-Filter add-ons can be used. The STATUS program is a sample of one such add-on.

    On a more useful scale, the following SRE-Filter add-ons can be obtained from the SRE-Filter home page.

    • A "news-group and list-server" discussion group package (FORUM.ZIP).
    • A full featured "web based bulletin board system" (BBS.ZIP)
    • A directory displayer, with user-settable display features (GETAFILE.ZIP)
    • A powerful, character-mode calculator with a web interface (CALCWWW.ZIP)
    Notes:
    • Ambitious programmers can look in the SRE-Filter manual for more information on writing to the SRE-Filter API. If you are really serious, you'll want to get the source code and documentation for SRE-Filter's macrospace library & independent threads.
    • If desired, you can suppress execution of CGI-BIN scripts and SRE-Filter add-ons, both across-the-board (the NO_PROCESSING parameter), and on a URL-specific basis (the NO_SSP permission).
    • SHOWDIR.DOC describes how to use the SHOWDIR dynamic directory processor (as an alternative to SRE-Filter's !DIR facility).

    Return to table of contents.

    
    
    
    ...... ...

    9)Uploads, HEAD requests, and other methods

    SRE-Filter supports some of the less frequently used HTTP 1.0 protocols, as well as a few proposed HTTP 1.1 protocols.
    The protocol Discussion

    HEAD requests Although not frequently used, HEAD method requests offer some throughput advantages when searching over a broad range of sites. To fully support such requests, SRE-Filter can parse the <HEAD> section of HTML documents, and return (in the response header) the contents of LINK, NAME and META-EQUIV elements. In fact, SRE-Filter can parse the <HEAD> for GET requests too (though this may not be a very useful activity).

    For further discussion, see the description of the AUTO_HEADER parameter in INITFILT.DOC, or see the SRE-Filter manual.


    File Uploads using HTML FORMS SRE-Filter's PUT_FILE facility supports the multipart/form-data method of file upload (as supported by NetScape 2.01). This requires the use of an HTML document that contains a special type of <FORM>.

    Notes:

  • For an example of such a document, you can examine UPLOAD.HTM .
  • The SRE-Filter FAQ contains a discussion of file uploads using HTML forms.
  • For further discussion, see the SRE-Filter manual.

  • Uploads from other servers Though somewhat obsolete with the advent of multipart/form-data method of file upload, SRE-Filter's GET_URL facility permits clients to transfer files from different http servers to your server.

    Notes:

  • For an example of the use of GET_URL, you can examine UPLOAD.HTM .
  • The UPLOAD_MINFREE and UPLOAD_MAXSIZE parameters (as described in INITFILT.DOC) can be used to control the size and extent of file upload capabilities.
  • For further discussion, see the SRE-Filter manual.

  • The PUT and DELETE methods The PUT and DELETE HTTP 1.1 methods allow clients to upload, and delete, files from your server. Most browsers do not support these methods.

    Notes:

    • The DOPUT.CMD and DODELETE.CMD files can be used to PUT and DELETE files from your server.
    • As a security measure, special URL-specific permissions must be granted before the PUT and DELETE methods are recognized.
    • For further discussion, see the SRE-Filter manual.

    Byte-range
    retrieval
    SRE-Filter supports byte-range retrieval. Byte-range retrieval, which consists of sending selected portions of a document, is used by Adobe Acrobat to retrieve selected pages from large .PDF files.

    For further discussion, see the description of the ACCEPT_RANGE parameter in INITFILT.DOC.


    Setting the expiration date of documents To work around NetScape's tendency to over-refresh documents that have an immediate expiration date, you can instruct SRE-Filter to modify GoServe's default temporary file response header. Specifically, you can change the immediate expiration date to an expire in 2 hours expiration date.

    The simple mode configurator's Miscellaneous section can be used to select this option. For further discussion, see the description of FIX_EXPIRE in INITFILT.DOC.

    Return to table of contents.

    
    
    
    ... ...... ...

    10)Pre-filter and Post-Filter Processing

    SRE-Filter does a few things before and after processing the request.
    Action Description

    Load Balancing SRE-Filter supports a very simple form of load balancing. Specifically, when the load on your server (measured in number of current transactions) get too high, SRE-Filter can redirect the request to (one of several) backup servers.

    For details, see the description of the BACKUPSERVERLIST and LOADTHRESHOLD parameters (in INITFILT.DOC), or see the SRE-Filter manual.


    Special Directives SRE-Filter understands several special directives. These are prefixes added to request strings that can modify SRE-Filter's logic (they all start with a ! )
    • !force/URL : Forces a logon (if needed), and suppresses the SSI-Cache
    • !norecord/URL : Do not record this request (in the RECORD_ALL_FILE)
    • !sendas_mimetype_mimesubtype/URL : Send the request with a content-type: mimetype/mimesubtype response header
    • !delsend/filename.ext : Return a transient file from (from the TEMPFILE_DIR directory), and then delete it.
    See the SRE-Filter manual for more details on these special directives.

    Pre-filter processing There may be occasions when you want some other filter to process a request before SRE-Filter get's a shot at it. For example, ambitous programmers may wish to implement a more sophisticated load balancing algorithim; or you might wish to implement an alternative access control mechanism. As a more practical example, SRE-Filter comes with PREFILTR.80; a pre-filter that passes a request to the GOREMOTE.80 server remote control package.

    The SRE-Filter manual describes how to enable pre-filters; or you can read the description of the PRE_FILTER and PREFILTER_NAME parameters in INITFILT.DOC.


    Post-filter processing After SRE-Filter has responsed to a request (and GoServe has closed the connection), you may wish to perform further actions. For example, you may wish to notify the webmaster of special events (see POSTMAIL.80 for an example), or you may wish to record select requests (see POSTRCRD.80 for an example).

    The SRE-Filter manual describes how to enable post-filters; or you can read the description of the POST_FILTER and POSTFILTER_NAME parameters in INITFILT.DOC.


    Using the RECORDALL_FILE SRE-Filter has a special built-in post-filter action: it can record all requests. This recording consist of augmenting a counter, it does not involve recording who made the request (you can always use the GOAUDIT.80 file for that).

    For details on SRE-Filter's record all requests option, see the SRE-Filter manual, or see the description of the RECORD_OPTION and RECORD_CACHE_LINES parameters in INITFILT.DOC.

    Return to table of contents.

    
    
    
    ...... ...

    Synopsis of Features

    The following summarizes most of SRE-FILTER's features: Return to the table of contents.
    
    
    
    
    
    ...... ...

    The Disclaimer

    In a nutshell:
    SRE-Filter, and related products, are to be used at your own risk; the authors assume no liability for any damage that may be caused by the use or misuse of these software products.
    The nice news is:
    SRE-Filter is copyright by Daniel Hellerstein, and hereby placed in the the public domain. You may use is it any manner you deem fit, up to and including use of the code in your own software.

    Out of the nutshell:
      Copyright 1996,1997 by Daniel Hellerstein. 
      Permission to use this program  for any purpose is hereby granted without fee, 
      provided that the author's name not be used in advertising or publicity
      pertaining to distribution of the software without specific written
      prior permision.
      With some exception, this includes the right to subset and reuse the code,
      with proper attribution. The exceptions are two fold:
         1)  Portions of the code adapted from other author's work
            (these are noted where appropriate); you'll need to contact these other
            authors for appropriate permissions.
        2)  SRE-Filter uses Quercus System's REXXLIB procedure library.  The
            license for REXXLIB gives the author the right to distribute REXXLIB
            without charge.  This right may NOT extend to redistributors.
            Please contact Quercus Systems for details.
            SRE-Filter also uses Info-Zip's UNZIPAPI.  Although this is freely
            available software, you may wish to contact Info-Zip for details.
      Furthermore you may also charge a reasonable re-distribution fee for 
      SRE-Filter; with the understanding that this does not remove the
      work from the public domain.
    
        THIS SOFTWARE PACKAGE IS PROVIDED "AS IS" WITHOUT EXPRESS
        OR IMPLIED WARRANTY.
        THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE PACKAGE,
        INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS.
        IN NO  EVENT SHALL THE AUTHOR (Daniel Hellerstein) OR ANY PERSON OR
        INSTITUTION ASSOCIATED WITH THIS PRODUCT BE LIABLE FOR ANY
        SPECIAL,INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER
        RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION
        OF CONTRACT,NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR
        IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE PACKAGE.
    
    
       SRE-FILTER was developed on the personal time of Daniel Hellerstein,
       and is not supported, approved, or in any way an official product 
       of my employer (USDA/ERS).
    

    Acknowlegments

    Though my wife probably put up with the most grief, the following people all contributed:

    Return to table of contents.

    
    
    
    ...... ...
    Do you have questions, comments, or suggestions relating to SRE-Filter? Send 'em in ( Daniel Hellerstein.) -- that's DANIELH@ECON.AG.GOV if your browser can't deal with MAILTO:.