First: Create a script. You can call it whatever you want. I will call it downloader.sh
.html
#!/bin/bash PROTOCOL="ftp" URL="server.example.com" LOCALDIR="/home/user/downloads" REMOTEDIR="dir/remote/server/" USER="user" PASS="password" REGEX="*.txt" LOG="/home/user/script.log" cd $LOCALDIR if [ ! $? -eq 0 ]; then echo "$(date "+%d/%m/%Y-%T") Cant cd to $LOCALDIR. Please make sure this local directory is valid" >> $LOG fi lftp $PROTOCOL://$URL <<- DOWNLOAD user $USER "$PASS" cd $REMOTEDIR mget -E $REGEX DOWNLOAD if [ ! $? -eq 0 ]; then echo "$(date "+%d/%m/%Y-%T") Cant download files. Make sure the credentials and server information are correct" >> $LOG fi
Second: Add it to crontab. If you want to execute it every exact 15 minutes inside an hour:linux
45,30,15,00 * * * * /home/user/downloader.sh >/dev/null 2>&1
If you want to execute it each 15 minutes no matter what is the starting minute:shell
*/15 * * * * /home/user/downloader.sh >/dev/null 2>&1
Explaining the variables:express
PROTOCOL
- What protocol to use. lftp
supports a good range of them: ftp, ftps, http, https, hftp, fish, sftp
and file
. https and ftps require lftp to be compiled with OpenSSL or GNU TLS support.URL
- Name or IP of the server. You can even add :PORT
at the end if your server doesn't use the default port of the protocol being used.LOCALDIR
- Where to save the files.REMOTEDIR
- Where to cd
on the remote server to get the files.USER
and PASSWORD
- ftp credentials.REGEX
- Regular expression to filter files to download. It can be useful if you want to download only files of a determined extension, for example. Use *
if you want to download everything.LOG
- Logfile location.Explaining some code logic:bash
1. - ifsession
if [ ! $? -eq 0 ]; then fi
The $?
variable is a special bash variable that means "status code of last command". Bash always return zero on successful command executions so, comparing -eq
(equal to) with the starting !
(negative) on an if
should be enough to see if cd
and lftp
had issues during execution. If you want a better log of what happened, you will have to crawl through those commands' documentation.app
2. - heredocside
lftp $PROTOCOL://$URL <<- DOWNLOAD DOWNLOAD
bash heredocs. It's a way to say "feed this command with this input list". I've named the limit string DOWNLOAD
so, everything between <<- DOWNLOAD
and DOWNLOAD
will be input to lftp
. You will see examples on the internet with <<
symbol but I prefer the <<-
version since it supports indentation.ui
3. - lftp commandsthis
user $USER "$PASS" cd $REMOTEDIR mget -E $REGEX
These are internal commands of lftp
that means respectively, auth the user with $USER
login and "$PASS"
password, change to $REMOTEDIR
and bulk download anything with the $REGEX
keywords. You can learn them by simply typing lftp
, and as soon as an lftp shell is opened, type ?
and press Enter or ? lftp-command-you-want
and press Enter. Example:
[root@host ~]# lftp lftp :~> ? !<shell-command> (commands) alias [<name> [<value>]] attach [PID] bookmark [SUBCMD] cache [SUBCMD] cat [-b] <files> cd <rdir> chmod [OPTS] mode file... close [-a] [re]cls [opts] [path/][pattern] debug [<level>|off] [-o <file>] du [options] <dirs> exit [<code>|bg] get [OPTS] <rfile> [-o <lfile>] glob [OPTS] <cmd> <args> help [<cmd>] history -w file|-r file|-c|-l [cnt] jobs [-v] [<job_no...>] kill all|<job_no> lcd <ldir> lftp [OPTS] <site> ln [-s] <file1> <file2> ls [<args>] mget [OPTS] <files> mirror [OPTS] [remote [local]] mkdir [-p] <dirs> module name [args] more <files> mput [OPTS] <files> mrm <files> mv <file1> <file2> [re]nlist [<args>] open [OPTS] <site> pget [OPTS] <rfile> [-o <lfile>] put [OPTS] <lfile> [-o <rfile>] pwd [-p] queue [OPTS] [<cmd>] quote <cmd> repeat [OPTS] [delay] [command] rm [-r] [-f] <files> rmdir [-f] <dirs> scache [<session_no>] set [OPT] [<var> [<val>]] site <site-cmd> source <file> torrent [-O <dir>] <file|URL>... user <user|URL> [<pass>] wait [<jobno>] zcat <files> zmore <files> lftp :~> ? mget Usage: mget [OPTS] <files> Gets selected files with expanded wildcards -c continue, resume transfer -d create directories the same as in file names and get the files into them instead of current directory -E delete remote files after successful transfer -a use ascii mode (binary is the default) -O <base> specifies base directory or URL where files should be placed
The knowledge related to know that mget
would be the right command inside lftp
came from reading manpages and searching for keywords like "bulk", "multi" or "mass", and knowing that the ftp(1)
command also have the mget
command so, probably lftp
would have an equivalent.
Manpage: lftp(1)