Download files from remote server using command-line: wget

This example shows how to download NYU VAGC galaxy and sky catalogue for SDSS DR7.

We want all the files from  http://sdss.physics.nyu.edu/datasweep/dr7 with the names *gal*.fits, *sky*.fits

  • Command line:
wget -b -r -np -nH --cut-dirs=2 -I datasweep/dr7 -A "*gal*","*sky*" http://sdss.physics.nyu.edu/datasweep/dr7
  • Explanation:

-b: run in background
-r: recursive retrieval of links and directories
-np: no parent, do not ascend to parent directories. Note this doesn't
work for absolute links to parent url.
-I: only retrieve the directory datasweep/dr7. use absolute path (the
full url without hostname).
-nH --cut-dirs=2: do not make server directory sdss.physics.nyu.edu
and cut-off following two levels of dirs datasweep/dr7 so that the
downloaded files are put directly in the current direcoty.
-A: include only files matching  *gal* and *sky*