From ns.itep.ru!CERN.ch!EU.net!newsfeed.internetmci.com!in2.uu.net!brighton.openmarket.com!decwrl!tribune.usask.ca!rover.ucs.ualberta.ca!hydra.chem.ualberta.ca!bruce Fri May 10 11:31:01 1996 Path: ns.itep.ru!CERN.ch!EU.net!newsfeed.internetmci.com!in2.uu.net!brighton.openmarket.com!decwrl!tribune.usask.ca!rover.ucs.ualberta.ca!hydra.chem.ualberta.ca!bruce From: bruce@news.srv.ualberta.ca (Bruce Clarke) Newsgroups: comp.os.os2.networking.www,comp.os.os2.networking.tcp-ip Subject: Re: Any good OS/2 FTP clients? Date: Fri, 03 May 1996 14:56:05 Organization: University of Alberta, Edmonton, Canada Lines: 355 Message-ID: <960503145605@hydra.chem.ualberta.ca> References: <4lqnil$ml3@newsgate.dircon.co.uk> Reply-To: Bruce.Clarke@ualberta.ca NNTP-Software: PC/TCP NNTP Xref: ns.itep.ru comp.os.os2.networking.www:8578 comp.os.os2.networking.tcp-ip:60275 In article <4lqnil$ml3@newsgate.dircon.co.uk> bob@gmt2000.co.uk (Bob Millar) writes: > I am looking for a good ftp package for OS/2. I can do everything below and much powerful things using PM Directory Match (PMDMATCH.EXE) as the front end, with Innoval's Surf'nRexx working as a back end. Since it is all programmable in Rexx, I can make it as fancy and as powerful as I want. > I presently use FTP-It which is superb but lacks drag'n'drop and the > ability to upload complete trees. Yes. I love FTP-It because it is multithreaded and can download a bunch of files simultaneiously at amazing speeds. > > It will be used for updating our website and needs to have the following > capabilities: > > 1) Uploading and downloading of files by drag'n'drop. FTP-It already does this. Another approach that gives you more control would be to set up an object that runs Surf'nRexx when a file is dropped on it. For example, lets say I want to mark a homework set where my 500 students have entered their answers into a form on my web server. I just drag and drop the "homework set 5" object on the "mark homework "wizard" and then everything is automatic. The Rexx script that runs uses Surf'nRexx go get data off serveral other computers, both by ftp and by runing CGI scripts on my web server. Then it marks the homework, and updates the web server so my students can view the results. > 2) Uploading and downloading whole directory structures by > drag'n'drop. This works great. I do it a number of ways. See below. > 3) Ability to create and delete directories. > 4) Ability to delete remote files. > 5) Ability to automatically determine ASCII/binary transfers. All the above work great. > 6) Ability to determine if a local file has changed and if not, not > to upload it. This is where you really need PMDirMatch. I have a lot of different stuff running, so I'll start with the simplest (oldest) and progress to the most complex (most recent). Method 1. Auto-unzipping a file which a daemon watches for ---------------------------------------------------------- I had this running 2 years ago. The root of the directory tree on my web server (running GoSERV) is called d:\gopher. I have a Rexx script that I run on the web server called WWWCRON.CMD. I have an object in the startup folder which starts WWWCRON whenever I reboot. WWWCRON sleeps all the time. But every 60 seconds it wakes up and checks for the file d:\gopher\wwwchem.zip. When this file appears it unzips it and erases it. To update the web server, the sending machine needs a mirror copy of the directory structure of the web server. Not all the files need to be present, just the ones that need to be uploaded. You need a Rexx script that changes to the directory that is equivalent to the one on the web server where the zip file will reside. Then the script must add the files to the ZIP file using Infozip with the -r switch. That will cause all files in the zip file to have paths that are relative to this location. When the file is unzipped, the files will all be placed in the correct locations in the data directory tree of the web server. I've been using this technique for about 15 months to upload our dept newsletter to the web server. A copy arrives on one machine by e-mail, and a bunch of web pages are created from it in html. These are then moved to the web server automatically by this technique. Method 2. Automating the creating of Mirror copies -------------------------------------------------- I have file called ZIPENT.CMD on my web server. It creates a zip file of the servers entire data directory tree, the tree where cgi scripts are kept, and the tree where the data is stored for the RexxBase software the server runs. On anther machine, I have a Rexx script that automatically creates a mirror copy using the zip file created by ZIPENT. This script uses Surf'nRexx to log on, and download the zip file. Then it deletes the old directory trees and unzips the zip file to create mirror copies of the directory trees. My home machine also contains a development version of the web server. I use PMDirMatch to compare the dev version with the real version, and copy files between them. Method 3. Automating directory matching and updates with PMDirMatch ------------------------------------------------------------------- This is my latest and greatest -- a general purpose set of Rexx scripts for syncronizing files and directory trees between machines. I've got a Warp machine at home and three OS/2 machines at work on the internet. I wanted to be able to use PMDirMatch to compare any part of any directory tree on any two machines. That lets me see files that are missing, or have the same name but are different times and dates, or even files that have the same name, time and date, but the bytes are different. PMDirMatch is a tremendously powerful directory comparison utility. I wanted to make it run across the internet so I could use it to compare directories on different machines and move files. Here how I've got it working. On each of my machines I run a Rexx script called PMDCRON.CMD. This script makes the machine a "ZIP" file server and also a "snapshot" server. The machine I'm using is the "client". It has to tell the remote server machine what it wants, and then the zip file or snapshot is created, moved to the client, and then I use PMDirmatch to compare files and do updates. I'll describe the server end and then the client end. The DirMatch Server ------------------- The server machines run PMDCRON.ZIP. This script iterates forever. On each iteration, it sleeps for 60 seconds and then checks for two files. It does further processing only if one or both of these files appear. It looks for files called AUTOZIP.GET or AUTOSNAP.GET. These files appear as a result of being uploaded by ftp. The files should have a single line with a complete path. For example, the homework sets on my office machine are in H:\TEACH\CH10296\HWMK and this might be in AUTOZIP.GET or AUTOSNAP.GET. If AUTOZIP.GET appears, PMDCRON.CMD reads the path, and zips the directory tree under the path. For example, if the path is H:\TEACH\CH10296\HMWK, the zip file is created from H:\TEACH\CH10296 and the directory HMWK is zipped with the -r switch. Thus, HMWK and all directories under it are included in the zip file. Eventually, the zip file will be moved to another machine and expanded from some directory. Suppose this is \TEMP. Expanding creates \TEMP\HMWK. The entire tree under HMWK on the server (sending) machine is recreated under \TEMP\HMWK on the client (receiving) machine. If AUTOSNAP.GET appears, PMDCRON uses a nice feature of PMDMATCH called a "snapshot". The snapshot is a file called AUTOSNAP.PMD and contains complete directory information -- much more than is available by ftp. For example, it contains EA sizes. In the example I'm using, the snapshot would be taken from H:\TEACH\CH10296\HWMK and would include all files in that directory and directories under it. Summarizing: Each of my machines runs PMDCRON.CMD, which turns it into a zip file and snapshot server. PMDCRON watches for AUTOSNAP.GET and AUTOZIP.GET and responds by creating AUTOZIP.ZIP or AUTOSNAP.PMD. I use the ZIP feature when the ZIP file will not be too large. I use the SNAP feature when I want to look at a lot of files, such as the entire hard drive. For example, a 200 MB hard drive with about 8000 files gives a snapshot file of about 1 MB. That is manageable, but a ZIP file of the entire hard drive would not be. The PMDirMatch Client --------------------- PMDirMatch has a screen like LapLink showing 2 directories side by side. You can copy and move files between the two sides. So the UI of PMDirMatch is just the perfect front end of a utility for comparing files and transferring them between machines. The really nice features of PMDirMatch is that it aligns files with the same filenames side by side. If a file is only in one of the directories being compared, there is a blank line on the side for the directory without the file. PMDM (PMDirMatch) has incredibly powerful file matching features. You can make it show all files, similar files (files with the same names), or unique files. PMDM has two a built in file comparison algorithms. You can choose which to use. One uses byte-by-byte comparisons of the files. The other uses time and date. PMDM decides whether files with the same name are identical or different by applying the file comparison method. If you tell it to display "similar files", it will display all files that are idential (exactly the same) plus files that have the same name but are differnet, either because they have different times and dates or because they are not exactly the same when compared byte-by-byte. If you tell it to display "different files", it will display all unique files plus files with the same name that are different. PMDM flags similar but different files in color. If you click the right mouse button on a pair of similar but different files, it pops up your favorite file comparison utility. I use PMDiff and that shows me the exact line by line differences. To use this as the front end for managing files on two machines across the internet, I set up a few Rexx scripts. I could make these work with drag and drop, but right now I just run them from the command line. The scripts that I use on my home machine to compare with my office machine (called hydra) are ZIPHYDRA.CMD and PMDHYDRA.CMD. Both of these prompt for two paths, one on the local machine and the other on the remote machine. I could make this drag and drop. I'd have to drop something that represents a folder on the remote machine as well as something that represents a folder on the local machine. Typing in a couple paths is easy and that's what I do. ZIPHYDRA.CMD checks that the local path is valid. Then it uploads AUTOZIP.GET with the remote machines path in that file. It waits 2 minutes. That allows 1 minute for the remote machine's PMDCRON to go through a complete iteration find AUTOZIP.GET, and still have a minute left to generate the ZIP file. Then ZIPHYDRA.CMD gets the ZIP file, and expands it in N:\TEMP. Finally it runs PMDirMatch so that it compares the local directory with the mirror copy. The other approach uses PMDHYDRA.CMD. This works essentially the same, except that the file it puts on the remote machine is AUTOSNAP.GET and the file it downloads is AUTOSNAP.PMD. It runs PMDirMatch so that the snapshot is loaded on one side and the local directory is loaded on the other side. This gives me a comparison of the files in the snapshot on the remote machine with the files in the matching directory (tree) on the local machine. Both the ZIP and SNAPSHOT approaches are very useful and have unique advantages. The ZIP approach lets me do byte-by-byte comparisons of files. Normally, ftp file transfers alter the times and dates on files and that is why I use byte-by-byte comparisons, rather than time and date. But when the ZIP method is used, the tree that is transferred contains all the same times and dates and even the same EA's as the remote machine. I use PMDM to update the local copy of files from the mirror copy. That gives me files with the same dates, times and EA's as the remote machine. The ZIP method is extremely good if I make changes to the same file at home and at work. I can see that they are different, and run PMDiff. Then I can merge the changes made and home and the changes made at work into a single file and save it on the local machine. The SNAPSHOT approach is really good for looking at the big picture, but it has a few problems. The author of PMDM (Larry Waibel) is working on enhancements that will solve those problems this week. PMDM always has a "current subdirectory" and only files and directories below the current subdirectory appear. You can change the current subdirectory up and down the directory tree by double clicking the ".." to move up to the parent, or double clicking a subdirectory name to change the current subdirectory to that subdirectory. This ability to navigate the directory tree does not work with snapshots but Larry is adding this ability right now. When it is working, you'll be able to load a snapshot of the entire hard drive of a remote machine and choose a current subdirectory somewhere on the snapshot. Then PMDM will only show the files and directories below that current subdirectory. To speed the navigation of the directory tree, Larry is adding a "directories only" view. This won't show any files, just the directory entries. When this is working, I'll be able to compare the complete directory structures on two machines, and focus in on any subtree using snapshots. If I want detailed byte-by-byte file comparisons, I can then use the ZIP method on that limited part of the directory tree. This software only requires 3 Rexx scripts. One is PMDCRON.CMD and the others are the client scripts ZIPXXX.CMD and PMDXXX.CMD, where XXX is the remote machine. The server machine has to be running FTPD and the client has to have some way to automate ftp transfers. Three of my machines handle the client end by completely different approaches. Nevertheless, the same PMDCRON and ZIPXXX.CMD and PMDXXX.CMD scripts work on all these machines. For FTP transfers (client end) one machine uses Surf'nRexx. Another machine automates the transfers using the NET.INI file in FTP Software's PC/TCP for OS/2. A third machine runs IBM's TCP/IP 2.0 and the transfers are automated using the NETRC file. I'm considering packaging up these Rexx scripts so others can use them. If you want them or think this is a good idea, please e-mail me. The scripts work very well for moving remote files to the client machine. I still need to do work to get the uploads working. Upload Issues ------------- Let's say that PMDM tells me I need to upload a file. I could find this out looking at a snapshot, or using the zip method where I look at a mirror copy of the remote tree. Within PMDM I can mark the local files, and "copy" them over to the snapshot side of the screen, or to the mirror copy if the zip method is being used. How can that operation cause these files to be uploaded to the correct locations? I am currently exploring several approaches. (1) PMDM allows the user to specify an "alternate directory" for copy operations. Using this feature would make it possible for all the files I need to upload to be copied to a separate directory tree that might reside under \TEMP\UPLOAD. This could then be zipped, uploaded by ftp, and expanded using the methods I've discussed. (2) PMDM allows the user to generate a CMD file instead of carrying out the move, copy and delete operations. The CMD file contains Rexx. A Rexx script could modify the CMD file so that the copy operations used (say) the FTPSEND.CMD program of Surf'nRexx to upload the files. (3) PMDM might get a Rexx interface. Larry is considering it. If he implements it, then when the user does a copy command inside PMDM, a user defined Rexx script could carry out the copy in whatever way the user wanted. The copy command could use ftp to move the file. This approach is the cleanest, because the operations between machines would occur while the user is using PMDM in a completely transparent way. In approaches (1) and (2) the files only get moved when a Rexx script is run after doing the copy inside PMDM. The methods I've outlined are not limited to OS/2 machines. There is a DOS program that takes snapshots. I also use InfoZip on to maintain a web site on a Sun the same way I do on my OS/2 machines. Because I've been using PMDM to compare files between Sun's and OS/2 machines, Larry has added many useful features to PMDM to help with the file comparisons. For example, the user of PMDM can specify whether the difference between OS/2's CR/LF line endings and Unix's LF line endings are to be ignored when doing byte-by-byte file comparisons. Larry Waibel, author of PMDirMatch deserves a prize for being the most responsive software developer I've every worked with. He fixes all sorts of small problems, sometimes within hours. For example, I once compared the CGI script directories between machines and found that PMDM was flagging idential Rexx scripts as "different" when it did a byte-by-byte comparison. Why? Rexx is compiled and stored in the files EA's. There were slight differences in the EAs. When I asked Larry to give me an "ignore EA's" option he provided me with a new beta 3 hours later. The PMDM server as I've described it only requires an FTP daemon on the server and ftp software that can be automated on the client. The client/server mechanism can be changed. If the server were running a web server, it could be made to operate with CGI scripts and that would not require a 2 minute wait for a response. All that I've described can be made to work with drag and drop eventually. But there is still a lot of work to do at the Rexx script level to get everything working completely transparently in both directions with all types of machines, with both the snapshots and zip file approaches. -- ------------------------------------------------------------ Bruce Clarke | mailto:Bruce.Clarke@UAlberta.ca Department of Chemistry | compuserve: 70740,3135 University of Alberta | ftp://hydra.chem.ualberta.ca Edmonton AB, T6G 2G2, Canada http://entropy.chem.ualberta.ca/faculty/clarke/home.htm