Package fr.ifremer.isisfish.simulator.launcher

Class SSHSimulatorLauncher

java.lang.Object
fr.ifremer.isisfish.simulator.launcher.SSHSimulatorLauncher
All Implemented Interfaces:
SimulatorLauncher
public class SSHSimulatorLauncher extends Object implements SimulatorLauncher
Use a remote simulation server. Data are uploaded into /scratch/isis-temp-4/ directory on FTP server. Then, on SSH side, this path is referenced as $SCRATCH/eftp/isis-temp-4. On the SSH server, the internal directory $SCRATCH/isis-fish-4 on not available threw FTP path, so we can't download files from it. ISIS-Fish must be installed on remote server (configuation option path). Datarmor file layout ($i = plan/as increment) :
  • /scratch/isis-temp-4/simulation-$id-preparation.zip : Simulation input zip file
  • /scratch/isis-temp-4/simulation-$id-prescript.bsh : Simulation prescript (optional)
  • /scratch/isis-temp-4/simulation-$id-script.seq : Qsub script
  • /scratch/isis-temp-4/simulation-$id-result.zip : Simulation output zip
  • /scratch/isis-temp-4/simulation-$id-result.zip.md5 : Simulation output zip md5 checksum
  • /scratch/isis-temp-4/simulation-$id-output.txt
  • /scratch/isis-temp-4/simulation-$id-pbs.id
  • /scratch/isis-temp-4/simulation-$id-pbs.out
  • /scratch/isis-temp-4/simulation-$id-pbs.err/li>
All /scratch/isis-temp-4/simulation-$id-* files are deleted on FTP server after result download. Special case :
  • standalone zip : /scratch/isis-temp-4/simulation-$shortid-result.zip (uploaded at first simulation)
  • standalone simulations : /scratch/isis-temp-4/simulation-$shortid-script.seq (uploaded at last simulation)
where $shortid is id of parent job (without increment), they are not deleted after result download.
Version:
$Revision$ Last update : $Date$ By : $Author$
Author:
chatellier
See Also:
  • JSch

  • Field Details

    • log

      protected static org.apache.commons.logging.Log log
      Class logger

    • freemarkerConfiguration

      protected freemarker.template.Configuration freemarkerConfiguration
      Freemarker configuration

    • QSUB_SCRIPT_TEMPLATE

      protected static final String QSUB_SCRIPT_TEMPLATE
      Freemarker qsub template.
      See Also:

    • REMOTE_TEMP_DIR

      protected static final String REMOTE_TEMP_DIR
      Nom du repertoire où sont stockés les fichiers temporaires.
      See Also:

    • SSH_TEMP_PATH

      protected static final String SSH_TEMP_PATH
      Path absolu du repertoire temporaire.
      See Also:

    • FTP_TEMP_PATH

      protected static final String FTP_TEMP_PATH
      Path d'upload sur le FTP qui pointe en fait sur $REMOTE_SSH_TEMP_PATH du point de vue SSH.
      See Also:

    • sshSession

      protected static com.jcraft.jsch.Session sshSession
      Opened session to ssh service. Stored in static context to not reask passphrase at each connection.

  • Constructor Details

    • SSHSimulatorLauncher

      public SSHSimulatorLauncher()
      Constructor. Init freemarker.

  • Method Details

    • initFreemarker

      protected void initFreemarker()
      Init freemarker configuration.

    • message

      protected void message(SimulationControl control, String message)
      Display message both in commons-logging and control text progress.
      Parameters:
      control - control
      message - message to display

    • maxSimulationThread

      public int maxSimulationThread()
      Description copied from interface: SimulatorLauncher
      Retourne le nombre maximal de thread de simulations simultanées supporté.
      Specified by:
      maxSimulationThread in interface SimulatorLauncher
      Returns:
      un nombre superieur a 0

    • getCheckProgressionInterval

      public int getCheckProgressionInterval()
      Description copied from interface: SimulatorLauncher
      Retourne l'intervalle de temps a utiliser entre deux verification de progresssion.
      Specified by:
      getCheckProgressionInterval in interface SimulatorLauncher
      Returns:
      intervalle (en secondes);

    • toString

      public String toString()
      Overrides:
      toString in class Object

    • simulationStopRequest

      public void simulationStopRequest(SimulationJob job) throws RemoteException
      Called if a started simulation has been ask to stop. Tell launcher to stop simulation if possible. Try to send a qdel command.
      Specified by:
      simulationStopRequest in interface SimulatorLauncher
      Parameters:
      job - job to stop
      Throws:
      RemoteException - Si pour l'execution de la simulation on utilise des resources distantes (serveurs) et que ceux si non pas pu etre contacte.

    • simulate

      public void simulate(SimulationService simulationService, SimulationItem simulationItem) throws RemoteException
      Execute une simulation. Dans le cas de ssh:
      • upload la simulation
      • construit le script pour qsub
      • upload le script qsub
      • ajoute le script a qsub
      Et :
      • lance le thread de control de la simulation
      Specified by:
      simulate in interface SimulatorLauncher
      Parameters:
      simulationService - le SimulationService qui a initie la simulation
      simulationItem - l'item contenant toutes les infos nécessaire au lancement de la simulation IMPORTANT Quoi qu'il arrive
      Throws:
      RemoteException - Si pour l'execution de la simulation on utilise des resources distantes (serveurs) et que ceux si non pas pu etre contacte. Dans ce cas la simulation n'a pas ete faite et cette exception est levee.

    • getSimulationStorage

      public SimulationStorage getSimulationStorage(SimulationService simulationService, SimulationControl control) throws RemoteException
      Retourne le storage où est stockée la simulation. Se connecte au serveur distant et télécharge les résultats de la simulation. Simulation must have been downloaded with updateControl(SimulationService, SimulationControl) before calling this method.
      Specified by:
      getSimulationStorage in interface SimulatorLauncher
      Parameters:
      simulationService - le SimulationService qui a initie la simulation
      control - le control de la simulation
      Returns:
      le storage contenant la simulation qui vient d'etre faite ou null en fait au lieu de faire une seul simulation, plusieurs ou ete faite par exemple tout un plan de simulation Cette classe est tout de meme responsable du stockage de chaque simulation dans des storages.
      Throws:
      RemoteException - Si pour l'execution de la simulation on utilise des resources distantes (serveurs) et que ceux si non pas pu etre contacte.

    • updateControl

      public void updateControl(SimulationService simulationService, SimulationControl control) throws RemoteException
      Met à jour la progression. Se connecte au serveur distant et télécharge le fichier de control. Injecte ensuite ce fichier dans le SimulationControl. Essaye aussi de telecharger le fichier md5 de la simulation, et, s'il est present, l'archive de résultat. Supprime tous les fichiers de la simulations apres avoir télécharger les résultats.
      Specified by:
      updateControl in interface SimulatorLauncher
      Parameters:
      simulationService - le SimulationService qui a initie la simulation
      control - le control de la simulation
      Throws:
      RemoteException - Si pour l'execution de la simulation on utilise des resources distantes (serveurs) et que ceux si non pas pu etre contacte.

    • getSSHSession

      protected com.jcraft.jsch.Session getSSHSession() throws com.jcraft.jsch.JSchException
      Get opened ssh session or try to open a new one. This method must synchronized.
      Returns:
      opened ssh session.
      Throws:
      com.jcraft.jsch.JSchException

    • openSSHSession

      protected com.jcraft.jsch.Session openSSHSession() throws com.jcraft.jsch.JSchException
      Connect to remote server throw SSH, and return session.
      Returns:
      valid opened session
      Throws:
      com.jcraft.jsch.JSchException

    • closeSSHSession

      protected void closeSSHSession(com.jcraft.jsch.Session session)
      Close ssh session.
      Parameters:
      session - session to close

    • getFtpClient

      protected org.apache.commons.net.ftp.FTPClient getFtpClient() throws IOException
      Open new FTP client connected to server and user loggued in.
      Throws:
      IOException

    • openFtpClient

      protected org.apache.commons.net.ftp.FTPClient openFtpClient() throws IOException
      Open new FTP client connected to server and user logged in.
      Throws:
      IOException

    • closeFTPClient

      protected void closeFTPClient(org.apache.commons.net.ftp.FTPClient ftpClient) throws IOException
      Close ftp session.
      Throws:
      IOException

    • putSshKeyOnRemoteServer

      protected void putSshKeyOnRemoteServer(com.jcraft.jsch.Session session, File sshKey)
      Add ssh key into $HOME/.ssh/authorized_keys file. Just connect and do an "echo xx >> .ssh/authorized_keys"
      Parameters:
      session - opened session
      sshKey -

    • uploadSimulationIfNecessary

      protected String uploadSimulationIfNecessary(org.apache.commons.net.ftp.FTPClient ftpClient, SimulationItem simulationItem, String simulationid, File simulationFile) throws IOException
      Upload simulation if necessary and always return the remote simulation zip path to use.
      Parameters:
      ftpClient - already open valid ftp session
      simulationItem - simulation item
      simulationid - simulation id
      simulationFile - simulation file to upload
      Returns:
      remote file path or null if errors
      Throws:
      IOException

    • uploadSimulation

      protected void uploadSimulation(org.apache.commons.net.ftp.FTPClient ftpClient, String remoteDirectory, String remoteSimulationZipPath, File simulationFile) throws IOException
      Perform simulation upload. Create remote temp directory if not exists.
      Parameters:
      ftpClient - already open valid ftp session
      remoteDirectory -
      remoteSimulationZipPath -
      Throws:
      IOException

    • downloadResultsArchive

      protected File downloadResultsArchive(org.apache.commons.net.ftp.FTPClient ftpClient, SimulationControl simulationControl, String md5sum) throws IOException
      Download simulation zip results. MD5 control check sum if done, return null, if checkSum fail. File if configured to auto delete at JVM shutdown.
      Throws:
      IOException - if download fail (can happen if remote file doesn't exist)

    • downloadResultsMD5File

      protected File downloadResultsMD5File(org.apache.commons.net.ftp.FTPClient ftpClient, String simulationId) throws IOException
      Download remote simulation md5 control file and store its content into temp file.
      Parameters:
      ftpClient - valid opened ftp session
      simulationId - id de la simulation
      Returns:
      downloaded temp file (file have to be manually deleted)
      Throws:
      IOException

    • downloadSimulationInformationFile

      protected File downloadSimulationInformationFile(org.apache.commons.net.ftp.FTPClient ftpClient, String simulationId) throws IOException
      Download remote information.
      Parameters:
      ftpClient - valid opened ftp session
      simulationId - id de la simulation
      Returns:
      downloaded temp file (file have to be manually deleted)
      Throws:
      IOException

    • clearSimulationFiles

      protected void clearSimulationFiles(org.apache.commons.net.ftp.FTPClient ftpClient, SimulationControl control) throws IOException
      Remove all $ISIS-TMP/simulation-$id-* files on datarmor.
      Parameters:
      ftpClient - valid opened ssh session
      control - simulation control
      Throws:
      IOException

    • uploadSimulationScript

      protected String uploadSimulationScript(org.apache.commons.net.ftp.FTPClient ftpClient, String simulationid, String simulationScript) throws IOException
      Upload script on remote server.
      Parameters:
      simulationScript - file to upload
      Throws:
      IOException

    • getResultArchiveName

      protected String getResultArchiveName(String simulationId)
      Get remote simulation zip name.
      Parameters:
      simulationId - simulation id

    • uploadPreScriptIfNecessary

      protected String uploadPreScriptIfNecessary(org.apache.commons.net.ftp.FTPClient ftpClient, String simulationId, String simulationPreScript) throws IOException
      Upload pre script on remote server. Return path if uploaded or null if no upload needed.
      Parameters:
      ftpClient - valid opened ftp session
      simulationId - simulation id
      simulationPreScript - script content
      Throws:
      IOException - if upload fail

    • startSimulation

      protected void startSimulation(com.jcraft.jsch.Session session, org.apache.commons.net.ftp.FTPClient ftpClient, SimulationItem simulationItem, String simulationid, String simulationRemoteZipPath, String remoteResultZip, String simulationPreScriptPath) throws Exception
      Start simulation if necessary. Current simulation can be started later with a PBS multi job.
      Parameters:
      session - ssh session
      simulationItem - simulation item (needed for additionnal info, simulation, number, indenpendant, etc...)
      simulationid - simulation id
      simulationRemoteZipPath - simulation preparation (input) zip path
      remoteResultZip - simulation result (output) zip path
      simulationPreScriptPath - simulation prescript
      Throws:
      Exception

    • getLaunchSimulationScriptContent

      protected String getLaunchSimulationScriptContent(String simuationId, String simulationZip, boolean standaloneZip, String simulationResultZip, String preScriptPath, boolean multipleSimulationScript) throws IOException
      Retourne un fichier temporaire contenant le script de lancement de simulation. Le fichier temporaire est configuré pour se supprimer tout seul.
      Parameters:
      simuationId - id de la simulation
      simulationZip - zip de la simulation
      standaloneZip - standalone simulation zip
      preScriptPath - simulation pre script path (can be null)
      multipleSimulationScript - if true build a multijob simulation script
      Returns:
      un Fichier temporaire ou null en cas d'exception
      Throws:
      IOException - if can't build script

    • getSimulationScriptLaunchContent

      protected String getSimulationScriptLaunchContent(String templateName, String simuationId, String simulationZip, boolean standaloneZip, String simulationZipResult, String preScriptPath, boolean multipleSimulationScript) throws IOException
      Utilise freemarker pour recuperer le contenu du script. Remplace aussi la variable $simulation du template.
      Parameters:
      templateName - url du template
      simuationId - id de la simulation
      simulationZip - zip de la simulation
      standaloneZip - standalone simulation zip
      simulationZipResult - zip resultat de la simulation
      preScriptPath - simulation pre script path (can be null)
      multipleSimulationScript - if true build a multijob simulation script
      Throws:
      IOException - if can't get script content

    • sendStartSimulationRequest

      protected void sendStartSimulationRequest(com.jcraft.jsch.Session session, String simulationId, String scriptRemotePath, int lastSimulationNumber) throws SSHException
      Add script in remote qsub queue.
      Parameters:
      session - valid opened session
      simulationId - simulation id (short version for a multiple job)
      scriptRemotePath - remote script path
      lastSimulationNumber - if >=0 start a multiple pbs job form 0 to lastSimulationNumber
      Throws:
      SSHException - if call fail

    • sendStopSimulationRequest

      protected void sendStopSimulationRequest(com.jcraft.jsch.Session session, String simulationId) throws SSHException
      Send qdel request on job.
      Parameters:
      session - valid opened session
      simulationId - simulation id
      Throws:
      SSHException - if call fail