In addition to serving as a discovery mechanism for community endpoints Globus enables users to create and (optionally) share their own endpoint definitions.
Logical endpoints can be created using the endpoint-add command. They can be continually modified (by adding physical addresses, renaming, etc.) and persist until explicitly deleted with the endpoint-remove command.
In the following example user demodoc adds an endpoint with a standalone ssh command. To demonstrate the Globus interactive shell mode, the user then adds two endpoints within an interactive Globus CLI session. Two logical endpoints are created, with vpac having one associated physical address and never having two:
$ ssh firstname.lastname@example.org endpoint-add vpac -p gsiftp://arcs-df.vpac.org:2811/ $ ssh email@example.com Welcome to globusonline.org, demodoc. Type 'help' for help. $ endpoint-add -p never-1.ci.uchicago.edu never $ endpoint-add -p never-2.ci.uchicago.edu never $ exit Connection to cli.globusonline.org closed.
Globus endpoint definitions are either public or private. Public endpoints are visible to all Globus users; private endpoints are visible only to those who created them. Here we see that after user demodoc makes an endpoint public, demodoc#never in visible in the public list:
$ ssh firstname.lastname@example.org $ endpoint-modify --public never Set 'never' to public $ endpoint-list -p alcf#dtn ci#pads go#ep1 go#ep2 demodoc#never nersc#dtn $ endpoint-list -p -v demodoc#never Name : demodoc#never Host(s) : gsiftp://never-2.ci.uchicago.edu:2811, gsiftp://never-1.ci.uchicago.edu:2811 Subject(s) : MyProxy Server: n/a
endpoint-list with no options displays the user's list of previously-activated endpoints (both public and private), along with the remaining activation time for each endpoint:
In addition to explicit creation, endpoints can be implicitly created by way of transfer and scp. If the transfer or scp command refers to a hostname instead of a logical name, a private endpoint will be automatically created to represent it. Further information about implicit endpoint creation can be found in the transfer and scpman pages.
Globus provides two commands for moving files: transfer and scp. The transfer command is the more feature-rich of the two; scp supports a well-known interface and is easy to use. Globus also supports features such as file synchronization and idempotent submission.
The following example shows a detached recursive scp. By default scp will be canceled if your ssh session is disconnected or you press Ctrl-C. However, Globus provides the -D option so you can create a detached scp task that runs in the background even if your ssh session is disconnected:
In contrast to scp, the transfer command reads an EOF or Ctrl-D terminated list of source and destination pairs from stdin and attempts to transfer all of the files in the list until successful or the user specified deadline has been reached. The following example directs Globus to recursively copy the contents of a directory from UChicago RCC to NERSC. It is the equivalent to the previous scp command, with the exception that any outstanding transfer requests not completed after the 6 hour deadline (-d 6h) will be ignored:
$ echo "ucrcc#midway/demodoc/sdata/10Kfiles100M/ nersc#dtn/project/mpccc1/dest/sdata/alcf20100122/ -r" | ssh email@example.com transfer -d 6h Task ID: 427b63ec-ee04-11df-aa30-1231350018b1 Created transfer task with 1 file(s)
Another way to specify a transfer dataset is via a file list. A file list can contain a mix of directory source/dest pairs and individual file source/dest pairs. The following example specifies that the 10,000 individual files listed in file 10Kmidway-nersc100MB.dat should be transferred:
$ cat ./10Kmidway-nersc100MB.dat | ssh firstname.lastname@example.org transfer Task ID: 28d854ae-ee18-11df-aa30-1231350018b1 Created transfer task with 10000 file(s)
The following two examples highlight the Globus one-way file synchronization feature. The first executes a file size-based check, the second executes a full md5sum check:
$ echo "go#ep1/share/godata/ go#ep2/~/ -r -s 1" | ssh email@example.com transfer Task ID: 609b53fc-ebff-11df-aa30-1231350018b1 Created transfer task with 1 file(s)
$ echo "ucrcc#midway/demodoc/sdata/10Kfiles100M/ nersc#dtn/project/mpccc1/dest/sdata/alcf20100122/ -r -s 3" | ssh firstname.lastname@example.org transfer Task ID: 1c05440a-ee57-11df-aa30-1231350018b1 Created transfer task with 1 file(s)
The following example demonstrates both inline endpoint creation and activation (note the automatically-generated private endpoint definition):
$ gsissh email@example.com scp -g alcf#dtn:~/samplefile.txt gridftp.lonestar.tacc.xsede.org:~/samplefile.txt Activating 'gridftp.lonestar.tacc.xsede.org:2811' Activating 'alcf#dtn:2811' Task ID: 3f4c2cc6-ee20-11df-aa30-1231350018b1 [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] 1/1 867.00 mbps $ ssh firstname.lastname@example.org endpoint-list *lone* -v Name : _gsiftp_gridftp.lonestar.tacc.xsede.org_2811 Host(s) : gsiftp://gridftp.lonestar.tacc.teragrid.org:2811 Subject(s) : MyProxy Server : n/a Credential Status : ACTIVE Credential Expires: 2010-11-12 16:47:16Z Credential Subject: /DC=org/DC=doegrids/OU=People/CN=Logan Detmers 319918/CN=1686987609/CN=1586129810 Name : xsede#lonestar Host(s) : gsiftp://tg-gridftp.lonestar.tacc.xsede.org:2811 Subject(s) : MyProxy Server : myproxy.xsede.org Credential Status : EXPIRED Credential Expires: 2010-11-12 05:56:46Z Credential Subject: /C=US/O=National Center for Supercomputing Applications/CN=Logan Detmers
$ ssh email@example.com transfer --generate-id 7f2fb1d6-ee76-11df-aa30-1231350018b1 $ cat ./10Kmidway-nersc100MB.dat | ssh firstname.lastname@example.org transfer --taskid=7f2fb1d6-ee76-11df-aa30-1231350018b1 Killed by signal 2. $ cat ./10Kmidway-nersc100MB.dat | ssh email@example.com transfer --taskid=7f2fb1d6-ee76-11df-aa30-1231350018b1 Deadline : 2010-11-12 19:24:31Z Task ID: 7f2fb1d6-ee76-11df-aa30-1231350018b1 Created transfer task with 10000 file(s) $ cat ./10Kmidway-nersc100MB.dat | ssh firstname.lastname@example.org transfer --taskid=7f2fb1d6-ee76-11df-aa30-1231350018b1 Notice: Task ID already created
Globus provides users with realtime and historical information about their tasks. Push mechanisms include email notifications of interesting events such as task completion, credential expiration, and account creation. Pull mechanisms return metadata at the task level (the task id returned by the scp and transfer commands) and the subtask level (each individual file transfer is considered a subtask and has a unique id.)
The default status command lists all pending tasks:
$ status Task ID : 28d854ae-ee18-11df-aa30-1231350018b1 Request Time: 2010-11-12 04:48:57Z Command : transfer (+10000 input lines) Status : ACTIVE
The status command also provides a way to list the last n tasks (-l n) regardless of state (-a):
$ status -l 4 -a Task ID : 3f4c2cc6-ee20-11df-aa30-1231350018b1 Request Time: 2010-11-12 05:46:51Z Command : scp -g alcf#dtn:~/samplefile.txt gridftp.lonestar.tacc.xsede.org:~/samplefile.txt Status : SUCCEEDED
Task ID : 28d854ae-ee18-11df-aa30-1231350018b1 Request Time: 2010-11-12 04:48:57Z Command : transfer (+10000 input lines) Status : ACTIVE
Task ID : 427b63ec-ee04-11df-aa30-1231350018b1 Request Time: 2010-11-12 02:26:30Z Command : transfer -d 6h (+1 input line) Status : SUCCEEDED
Task ID : 4a3c471e-edef-11df-aa30-1231350018b1 Request Time: 2010-11-11 23:56:24Z Command : scp -D -r ucrcc#midway:/demodoc/sdata/10Kfiles100M/ nersc#dtn:/project/mpccc1/dest/sdata/alcf20100122/ Status : SUCCEEDED
The default details command provides an overview of a transfer’s state:
$ details 28d854ae-ee18-11df-aa30-1231350018b1 Task ID : 28d854ae-ee18-11df-aa30-1231350018b1 Task Type : TRANSFER Parent Task ID : n/a Status : ACTIVE Request Time : 2010-11-12 04:48:57Z Deadline : 2010-11-13 04:48:57Z Completion Time : n/a Total Tasks : 10000 Tasks Successful : 8831 Tasks Expired : 0 Tasks Canceled : 0 Tasks Failed : 0 Tasks Pending : 1169 Tasks Retrying : 8 Command : transfer (+10000 input lines) Files : 10000 Directories : 0 Bytes Transferred: 925997465600 MBits/sec : 2224.619
The details -t command lists subtasks (i.e. individual files) for an scp or transfer task. In the following example the command produces a 10,001 line file (a header, plus one line for each file):
The events command provides information about events that occurred while executing a task. In this first example user demodoc is inspecting the progress of an earlier checksum-based sync by examining the "files_summed=" counts:
=== Task Details === Task ID : 28d854ae-ee18-11df-aa30-1231350018b1 Task Type : TRANSFER Parent Task ID : n/a Status : SUCCEEDED Request Time : 2010-11-12 04:48:57Z Deadline : 2010-11-13 04:48:57Z Completion Time : 2010-11-12 05:51:08Z Total Tasks : 10000 Tasks Successful : 10000 Tasks Expired : 0 Tasks Canceled : 0 Tasks Failed : 0 Tasks Pending : 0 Tasks Retrying : 0 Command : transfer (+10000 input lines) Files : 10000 Directories : 0 Bytes Transferred: 1048576000000 MBits/sec : 2248.957
The cancel command enables you to kill pending transfers for a given task. Files already copied by Globus are unaffected by cancel. Information about the state of each file can be extracted with details (SUCCEEDED files were transferred prior to the cancel):