Korn shell hack of the day

or, "Scope creep and you, a typical example of the modern IT project"

Sometimes I find myself knee deep in a complicated shell script that probably would have been better off written in perl or Java. Today was one of those days. What started as a simple script I wrote for managing sftp traffic to a couple vendors has been transmogrified to handle calling Glub to do an ftps transfer (the "other" secure FTP) from a vendor, then delivering the files to an internal Windows share drive. So far so good, but feature bloat ended up straining my creativity to tackle each new problem, and ultimately saddled me with a workable solution that, while interesting, is sort of a stinker.

The connection to the Windows share was supposed to be via a mounted drive. I would copy the files to a Unix directory, and Unix would manage the SMB transfer transparently. This turned out to be unsupported by our data services group unless the Windows server in question was virtual. It was not. So suddenly I'm on the hook for adding SMB support to the process.

A quick search revealed the JCIFS library, which I was able to turn into a Java class that takes commandline inputs for destination server and path, and reads credentials from a config file. Just call "java ToShare" instead of Unix's "cp", and Bob's your uncle. Of course, the script has to grow a little bit to handle the extra config variables, where to find java, and what the classpath should be.

Second, the Windows guy was having problems automating unzipping the files before feeding them to his app. For whatever reason, the approach we settled on was to have my script unzip the files, and send him the artifacts AND the original .zip files. So the script needed to grow in complexity, managing unzipping and iterating through the unzipped artifacts, and the source archives, and sending each over my java SMB hack. Which looks like this:

# Simple log function. Echo the date and passed message to the logfile
logit(){
  echo "`date` $1" >> $LOGFILE
}

# If there's an error, call this, which sends a notification email and exits
error_handler() {
  echo "$1" | mailx -s"FTPS receive script failure" $ERROR_EMAIL_ADDRESS
  logit "$1"
  exit 1
}

# Error checker which run after most commands, if the return code from the previous command
# isn't 0, give the error message specified to error_handler.
check_error() {
  if [ $? -gt 0 ]
  then
    error_handler "$1"
  fi
}

# Find all the files we downloaded, and unzip them, sending output to the logfile
find $LOCAL_DIR/* -prune -name "$REMOTE_FILE_PATTERN" -exec unzip -d $LOCAL_DIR {} >> $LOGFILE \;
check_error "unzip failed"

# Output from unzip will contain either "inflating" or "extracting" for each file in the archive.
# Look for these messages in the log file, find the filename part, and iterate over them
for file in `grep -e inflating: -e extracting: $LOGFILE | awk '{print $2}'`
do
  # Run the SMB program to send the file to the Windows share...
  $SMB_CMDLINE ToShare $file $SMB_SERVER $SMB_PATH
  check_error "Failed to copy $file to $SMB_SERVER"
  # ...and then delete it. This is fine because the original zipped files are untouched.
  rm $file
  check_error "Failed to delete $file"
done

# Next, do the same thing for the zipped files themselves...
for file in `find $LOCAL_DIR/* -prune -name "$REMOTE_FILE_PATTERN"`
do
  $SMB_CMDLINE ToShare $file $SMB_SERVER $SMB_PATH
  check_error "Failed to copy $file to $SMB_SERVER"
  # ...except just move the files to an archive directory
  mv $file $LOCAL_ARCHIVE_DIR
check_error "Failed to archive $file"
done
logit "Files successfully copied to $SMB_SERVER, moving files on remote server to archive directory"

After adding this functionality, I was cursing my desire to do this in shell instead of writing a real program. But as these things go, the project's timeline was short, and starting over was out of the question. Plus, as I'm wont to do, I gave a lot of assurances that a file transfer script was a no-brainer before I had all the facts - one of the facts being that the vendor was using ftps, not sftp as advertised, the whole reason I chose this script template in the first place.

Third, the vendor requested that after downloading files, that they be moved to an archive directory on the vendor's ftps server. Normally it's polite to delete a file after you download it from a vendor, as it saves them space requirements, and can also let them know at a glance what files have been pulled. Moving a file to another directory is asked for rarely, but usually isn't a problem. For this process, however, multiple files would be available at once, all of which had timestamps in the filename instead of fixed names. A tolerable approach would have been to download and delete using mget and mdel, but there is no facility in standard ftp clients for batch renames on the remote server.

Had I started with perl, I could have played with named pipes and searching logfiles for filenames to build a set of rename commands to print to the pipe. Had I started with Java, I could have used the Glub bean and managed the ftps session on the fly, getting a file list and iterating over it with RETR and RNFR/RNTO commands. And there wasn't time to do either. And the Unix "expect" program wasn't installed, so I couldn't manage the ftps session that way. No, I was unfortunately going to have to do this in two sessions. Workable, but ugly, and it's generally considered bad mojo to log on to a server twice in the same process. But that's what I did.

Fortunately, Glub logs all the server RETR commands, giving me something simple to grep for with which to build a list:

# Build the multiple rename (mren) string. For each logfile line starting with "RETR",
# print out a corresponding line that says "rename (file) archive/(file)", save as $MREN
export MREN=`perl -lne 'if (/^RETR /){s!RETR (.+)!rename $1 $ENV{REMOTE_ARCHIVE_DIR}/$1!;print}' $LOGFILE`

# Log on to remote ftps server, execute $MREN lines
$JAVA -Dglub.user.dir=$HOME -Duser.dir=$FTPS -jar $FTPS/secureftp2.jar <<EOF
log $LOGFILE
open $FTPS_HOST $FTPS_PORT
user $FTPS_USER $FTPS_PASS
cd $REMOTE_DIR
$MREN
bye
EOF
check_error "ftps archive session failed"

logit "Session successful"

So eventually I'll be able to circle around and write this in a more sensible manner, or better yet, convince the vendor that they're better off if I just delete the files, and the Windows people that unzipping is really a breeze. The good news to all this madness is that I added about 10% to my commandline Unix bag of tricks, and you really can do a lot of neat stuff with shell scripts.