GT 6.0 Release Notes: GridFTP

indexterm:[release notes (GridFTP)]


1. Component Overview

GridFTP is a high-performance, secure, reliable data transfer protocol optimized for high-bandwidth wide-area networks. The GridFTP protocol is based on FTP, the highly-popular Internet file transfer protocol. We have selected a set of protocol features and extensions defined already in IETF RFCs and added a few additional features to meet requirements from current data grid projects.

1.1. Feature Summary

Features new in GT 6.0:

  • Wider support for UDT

Features that continue to be supported from previous versions

  • Chrooting GridFTP server
  • Synchronize datasets
  • Improved failure restart capability in globus-url-copy
  • Stall detection
  • Load balancing in globus-url-copy
  • GridFTP over UDT
  • SSH security for GridFTP control channel
  • Running the GridFTP server with GFork GridFTP
  • Multicasting / Network overlays (EXPERIMENTAL)
  • Netlogger’s bottleneck detection for GridFTP transfers (EXPERIMENTAL)
  • GSI security: This is the PKI based, de facto standard security system used in Grid applications. Kerberos is also possible but is not supported and can be difficult to use due to divergence in the capabilities of GSI and Kerberos.
  • Third-party transfers: Very common in Grid applications, this is where a client mediates a transfer between two servers (both likely at remote sites) rather than between the server and itself (called a client/server transfer).
  • Cluster-to-cluster data movement or Striping: GridFTP can do coordinated data transfer by using multiple computer nodes at the source and destination.
  • Partial file access: Regions of a file may be accessed by specifying an offset into the file and the length of the block desired.
  • Reliability/restart: The receiving server periodically (the default is 5 seconds, but this can be changed) sends restart markers to the client. This marker is a messages specifying what bytes have been successfully written to the disk. If the transfer fails, the client may restart the transfer and provide these markers (or an aggregated equivalent marker), and the transfer will pick up where it left off. This can include holes in the file.
  • Large file support: All file sizes, lengths, and offsets are 64 bits in length.
  • Data channel reuse: Data channel can be held open and reused if the next transfer has the same source, destination, and credentials. This saves the time of connection establishment, authentication, and delegation. This can be a huge performance difference when moving lots of small files.
  • Integrated instrumentation (Performance Markers).
  • Logging/audit trail (Extensive Logging in the server).
  • Parallel transfers (Multiple TCP streams between a pair of hosts).
  • TCP Buffer size control (Protocol supports Manual and Automatic; Only Manual Implemented).
  • Server-side computation (Extended Retrieve (ERET) / Extended Store (ESTO) commands).
  • Based on Standards: RFC 959, RFC 2228, RFC 2389, IETF Draft MLST-16 , GGF GFD.020.

Other Supported Features

  • On the client side we provide a scriptable tool called globus-url-copy. This tool can take advantage of all the GridFTP protocol features and can also do protocol translation between FTP, HTTP, HTTPS, and POSIX file IO on the client machine.
  • We also provide a set of development libraries and APIs for developers wishing to add GridFTP functionality to their application.

Deprecated Features

  • None

1.2. Summary of Changes in GridFTP

1.2.1. New Features: GridFTP

  • Highlights

    • UDT Support for Wide-Area, Lossy Networks
    • UDT NAT Traversal, supported by Globus Connect Personal for
    • S3/HTTP support, allowing GridFTP transfers to and from Amazon S3
    • DCSC for cross-domain transfers without mutual trust
  • Other New Features

    • None

1.2.2. Improvements: GridFTP

  • Improved testing

1.3. Fixed Bugs for GridFTP

  • Fix crash on DCAU N with custom net stack

1.4. Known Problems in GridFTP

  • GRIDFTP-178: Hanging server processes with GT5.0.4 striped GridFTP servers
  • GT-2: GUC will treat directory as if its a file for the purpose of sync comparison
  • GT-4: Can’t build gridftp server with alternate CC, needed by LTA
  • GT-5: globus-url-copy doesn’t pipeline url input from a file
  • GT-6: GridFTP installation on OpenBSD 4.9
  • GT-7: Issues with netlogger style logs
  • GT-8: globus-url-copy -rst … -st … segfault if the options are not correctly synchronized
  • GT-10: Problem checking directory permissions with MLST
  • GT-11: globus-url-copy -rst segfault if the network is not working
  • GT-12: GUC exits with zero when it hits a stall-timeout
  • GT-273: can discover user names even with restrict path
  • GT-284: GridFTP pipelining and reliability issues
  • GT-318: globus-url-copy (guc) and "-do" option inconsistency
  • GT-374: Can’t share files in a path structure with symlinks: path not allowed error
  • GT-397: gridftp doesn’t error if file with unrepresentable name is used (MLSC)
  • GT-413: Clarify errors whenever sharing file operations fail
  • GT-414: globus-gridftp-server closing connection before returning cksm value
  • GT-428: Improve handling of hanging GridFTP server processes
  • GT-450: extra newline in PASV error message
  • GT-472: GridFTP server fails to detect client disconnection with piplining

1.5. Technology dependencies

GridFTP depends on the following GT components:

  • Non-WS (General) Authentication & Authorization
  • C Common Libraries
  • XIO

GridFTP depends on the following 3rd party software:

  • OpenSSL (version is included in release)

1.6. Tested platforms

Table 1. Tested Platforms

Operating System Distribution Version(s) Architecture(s)

Linux

CentOS

5, 6

i386, x86_64

7

x86_64

Fedora

20, 21, 22

i386, x86_64

Red Hat Enterprise Linux

5, 6

i386, x86_64

7

x86_64

Scientific Linux

5, 6

i386, x86_64

7

x86_64

SUSE Linux Enterprise Server

11SP3

x86_64

Debian

6, 7, 8

i386, amd64

Ubuntu

12.04LTS, 14.04LTS, 14.10, 15.04

i386, amd64

Mac OS X

10.6-10.10

i386, x86_64

Solaris

OmniOS

r151006

x86_64

Windows 7

Cygwin

i386, x86_64

MingW64

i386, x86_64


While the above list includes platforms on which we have tested GridFTP, it does not imply support for a specific platform. However, we are interested in hearing reports of success or bug reports on any platform.

1.7. Backward compatibility summary

Protocol changes since GT 5.2

  • None

API changes since GT 5.2

  • None

Exception changes since GT 5.2

  • Not Applicable (GridFTP is not Java-based)

Schema changes since GT 5.2

  • Not Applicable (GridFTP is not SOAP-based)

1.8. Associated Standards

Associated standards for GridFTP:

2. For More Information

See GridFTP for more information about this component.