Data Transfer, Import, Handling, and Formatting

This BioCompanion copy is a demo version . This section is intended for the interested researcher who works on more than one system, and wants to get familiar with the transfer procedures to analyse data of various origin by molecular biology computing programs. This session assumes that you have access to your desktop system and one or more other computers in a networked environment.

Transfer of Data in between Computers

In order to work with sequence data that you have transferred from another computer, you need to know what a sequence format is. The procedure to type sequence data manually into files is described in section "Sequence Editing" .

Note that word processors on personal computers quite frequently store data in non-ASCII format. Make sure that the file you want to transfer is really plain text. If needed, use the <save as> option and select <printed text> or <printed text with line breaks>.

'ftp'

The file transfer program (ftp) requires that the computers which shall exchange data run the same protocol, the TCP/IP (Internet Protocol) suite. Your computer has to be configured accordingly with the appropriate software .

Personal Computer Setups

To find the 'ftp' program on your desktop computer, search for the suitable application on your hard disk or ask your colleagues how to get the required software. Ask your system manager if you have questions about the configuration of your network . The big advantage of the personal computer setups is the user-friendly interface. All 'ftp' applications follow the same scheme:

Open communication as usual : supply host name, user name, and password (even for anonymous ftp (see below) you will ALWAYS need these three.)

Select the working directory at the remote host

Select the file(s) to be transferred

Select the receiving directory at the local host

Take care to transfer files in the correct mode. Sequence data files are usually text-only files and, therefore, need to be transferred in text mode (also called ASCII mode).

NOTE: Personal computers with modern operating systems (Macintosh, Windows 95 or NT, OS/2) allow file names which cannot be used on UNIX or VMS systems (which are usually installed at the host of interest).

Bear in mind the following rules for file names in order to facilitate working:

Use common case in all names (e.g., all-lowercase).

Do not use special characters and blanks; useful separators of letters are hyphen (-) and underscore (_).

Do not use numbers as first character, rather start with an alphabetic character.

Use file names of sensible length. More than 8 characters are difficult to type if you work with command line interfaces. On the PC (Win 3.x or DOS), long file names are not possible.

Use extensions as suggested by the program package you work with. E.g., seq and pep for sequences, and dat or txt for data and text files.

From and to other Hosts via 'ftp'

If you do not want to use the graphical user interface on the PC or Mac, or if you cannot do so because you want to transfer data in between other hosts, you need to call the ftp program on the command line of one host and transfer data to the other host. The details of this procedure depend on the implementations of the 'ftp' program in UNIX, VMS, or other operating systems. In general, the following points are important:

The style of the remote file names reflects the operating system of the remote host.

UNIX-like navigation is generally possible, but the navigation of the local host is usually better supported.

File names may not violate the conventions of either local or remote operating system characters, case, length and other details. Specifically, the "version numbers" of the VMS operating system and special characters (symbols, blanks, etc.) of personal computer systems must be considered as potential traps.

The following table gives an overview on the most important 'ftp' commands:

 
   
what                           | 'ftp' command   
-------------------------------+-------------------   
see where you are              | pwd   
-------------------------------+-------------------   
look for files                 | ls    
-------------------------------+-------------------   
change to subdirectory "test"  | cd test   
-------------------------------+-------------------   
go one level up                | cd ..   
-------------------------------+-------------------   
set file type binary           | bin   
-------------------------------+-------------------   
set file type text             | ASCII   
-------------------------------+-------------------   
get file "t1.seq"              | get t1.seq   
-------------------------------+-------------------   
get all files "t1.*"           | mget t1.*   
-------------------------------+-------------------   
transfer "t1.seq" to remote    | put t1.seq   
-------------------------------+-------------------   
toggle question in 'mget' mode | prompt   
-------------------------------+-------------------   
print progress during transfer | hash

Note that the use of FTP requires that you provide user name and password in order to access the remote directory.

SECURITY ADVICE: You should never leave your terminal or PC unattended if you are logged in to the computer. To ensure data security, you should avoid using other people's accounts. FTP access to a remote computer implies full read/write access to the remote data and is as sensitive as login via telnet or similar.

SECURITY ADVICE: File transfer of own data to another host implies that you give away your data to an environment which is potentially insecure as you do not know how security is handled on the remote system. Be aware that the unsolicited data export of any data with any tool, such as ftp, electronic mail or other, will be a potential risk to your success if these data are non-public data.

The mechanism to use anonymous ftp is important for data retrieval without the need to give you a dedicated username/access combination on any potentially interesting host on the networks. The Internet provides hundreds or thousands of hosts which store biologically relevant data. Be aware that the recent enthusiasm for the World Wide Web (WWW) has superseded the use of anonymous ftp in many areas - information access to small files is by now handled with WWW in much more elegant fashion. If you watch out for the actual identifier of the document in WWW you might still see the address in a format like ftp://host.name.site.net or similar which is an indication that your WWW browser actually utilises anonymous ftp to get the information of interest.

In order to access a remote host via anonymous ftp, it is requires that the server of choice is specifically configured for this option:

 
                host name              username           password     
-----------------------------------------------------------------------------  
normal use      any multi-user         your user name     your password  
  
anonymous ftp   ftp server             anonymous          your eMail address

NOTE: Not all sites will allow that individuals configure 'anonymous FTP' on any machine but a certain policy is in place which defines the desired and allowed activities on the network.

SECURITY ADVICE

(1) If you configure your PC or Macintosh for anonymous FTP usage on temporary or other basis, make sure that the directory which is accessible for the user 'anonymous' is NOT THE MAIN HARD DISK directory (on DOS, typically C:, on Macintosh, typically HD-500 or whatever the name of the hard disk is). If you fail to ensure that 'anonymous' gets an own, authorised directory , you risk data security and license ramifications.

(2) Some PC or Macintosh setups will automatically configure anonymous FTP for you or will suggest that you do so upon installation. Turn it OFF unless you really need it for scientific data exchange. Refer to the Internet access policy if you are in doubt whether you are allowed to configure your desktop at all.

(3) If you are a maintainer of an FTP archive, make sure that anonymous FTP directories are never writable unless you apply very tight security on permission based on the host name. You are responsible for anonymous FTP file server contents and must not allow the deposition of illegal or voluminous material on your desktop computer.

DECnet Copy

The option to copy files via DECnet is not described here for the sake of brevity.

SECURITY ADVICE: You should never use this mechanism in automatic procedures to avoid that your password becomes visible in stored files unless you use the PROXY mechanism, which is a potential security breach as well. Make sure that no one peeks at your screen while using DECnet copy - the ftp program (see above) is your only safe option.

Remote UNIX Copy

The option to copy files via the command 'rcp' is not described here for the sake of brevity.

SECURITY ADVICE: You should never use the 'trusted hosts' mechanism to avoid that you need to supply your password. Unless you run a 'yellow page' or 'network information system' (NIS) the ftp program (see above) is your only option.

'Kermit'

If you are connected via a serial line (i.e., rather old networking or modem lines), you may need to use the program 'Kermit'. On the remote computer you must give the following command to receive a data file:

% kermit -r test.seq

The procedures are reversed if you want to transfer a data file from the remote to the local computer. On the remote computer you need to type the following command to send a data file:

% kermit -s test.seq

The local options are reversed. On MS-DOS you must start with the local escape character which is shown at the bottom of the screen.

<CTRL><A> c

Then, you can give the command to send (or receive) a file, e.g.,

KERMIT-MS> send test.seq

To get back, type

KERMIT-MS> connect

NOTE: The local escape character might vary (e.g., <CTRL><[> c), but is usually shown at the bottom line of the screen.

On Macintosh and other Graphical User Interfaces there are usually options in the <File> or similar menu.

ZMODEM

This method is known from bulletin boards and other servers, but rarely used in molecular biology environments. It covers file transfer with built-in compression and its use is similar to the 'kermit' program.

JAMF source file: file.jam
Next file in HTML: 'File Handling Commands on Various Operating Systems '

[next page] , or [overview] , or [table of contents]