public class Cluster extends Bundle
- stagein is also clustered/bundled per level. In Bundle it was for the whole workflow. - keys that control the clustering ( old name bundling are ) cluster.stagein and cluster.stageoutIn order to use the transfer refiner implemented by this class,
- the property pegasus.transfer.refiner must be set to value Cluster
Bundle.BundleValue, Bundle.PoolTransfer, Bundle.TransferContainer
Modifier and Type | Field and Description |
---|---|
static java.lang.String |
DEFAULT_LOCAL_STAGE_IN_CLUSTER_FACTOR
The default clustering factor that identifies the number of transfer jobs
that are being created per execution pool for stageing in data for
the workflow.
|
static java.lang.String |
DEFAULT_LOCAL_STAGE_OUT_CLUSTER_FACTOR
The default bundling factor that identifies the number of transfer jobs
that are being created per execution pool for stageing out data for
the workflow.
|
static java.lang.String |
DEFAULT_REMOTE_STAGE_IN_CLUSTER_FACTOR
The default clustering factor that identifies the number of transfer jobs
that are being created per execution pool for stageing in data for
the workflow.
|
static java.lang.String |
DEFAULT_REMOTE_STAGE_OUT_CLUSTER_FACTOR
The default bundling factor that identifies the number of transfer jobs
that are being created per execution pool for stageing out data for
the workflow.
|
static java.lang.String |
DESCRIPTION
A short description of the transfer refinement.
|
private int |
mCurrentSILevel
The current level of the jobs being traversed.
|
protected java.util.Map<java.lang.String,Bundle.PoolTransfer> |
mStageInLocalMapPerLevel
A map indexed by site name, that contains the pointer to the stage in
PoolTransfer objects for that site.
|
protected java.util.Map<java.lang.String,Bundle.PoolTransfer> |
mStageInRemoteMapPerLevel
A map indexed by site name, that contains the pointer to the symlink stage
in PoolTransfer objects for that site.
|
private java.util.Map<java.lang.String,Job> |
mSyncJobMap
Maps the site name to the current synch job
|
DEFAULT_LOCAL_STAGE_IN_BUNDLE_FACTOR, DEFAULT_LOCAL_STAGE_OUT_BUNDLE_FACTOR, DEFAULT_REMOTE_STAGE_IN_BUNDLE_FACTOR, DEFAULT_REMOTE_STAGE_OUT_BUNDLE_FACTOR, mAddNodesForSettingXBit, mJobPrefix, mPegasusProfilesInProperties, mSetupMap, mSiteStore, mStageinLocalBundleValue, mStageInRemoteBundleValue, mStageOutLocalBundleValue, mStageOutRemoteBundleValue
mCreateRegistrationJobs, mFileTable, mLogMsg, mPPS, mRelationsMap
mDAG, mLogger, mPOptions, mProps, mRemoteTransfers, mTPT, mTXInterImplementation, mTXStageInImplementation, mTXStageOutImplementation, mTXSymbolicLinkImplementation, mXMLStore
INTER_POOL_PREFIX, LOCAL_PREFIX, REGISTER_PREFIX, REMOTE_PREFIX, STAGE_IN_PREFIX, STAGE_OUT_PREFIX
Constructor and Description |
---|
Cluster(ADag dag,
PegasusBag bag)
The overloaded constructor.
|
Modifier and Type | Method and Description |
---|---|
void |
addStageInXFERNodes(Job job,
boolean localTransfer,
java.util.Collection files,
int jobType,
java.util.Map<java.lang.String,Bundle.PoolTransfer> stageInMap,
Bundle.BundleValue cValue,
Implementation implementation)
Adds the stage in transfer nodes which transfer the input files for a job,
from the location returned from the replica catalog to the job's execution
pool.
|
void |
addStageInXFERNodes(Job job,
java.util.Collection<FileTransfer> files,
java.util.Collection<FileTransfer> symlinkFiles)
Adds the stage in transfer nodes which transfer the input files for a job,
from the location returned from the replica catalog to the job's execution
pool.
|
protected void |
constructCondorKey(Job job,
java.lang.String key,
java.lang.String value)
Constructs a condor variable in the condor profile namespace
associated with the job.
|
private Job |
createSyncJobBetweenLevels(java.lang.String name)
It creates a NoOP synch job that runs on the submit host.
|
void |
done()
Signals that the traversal of the workflow is done.
|
protected java.lang.String |
getComputeJobBundleValue(Job job)
Returns the bundle value associated with a compute job as a String.
|
java.lang.String |
getDescription()
Returns a textual description of the transfer mode.
|
protected int |
getSISiteBundleValue(java.lang.String site,
java.lang.String deflt)
Determines the bundle factor for a particular site on the basis of the
stage in bundle value associcated with the underlying transfer
transformation in the transformation catalog.
|
protected Bundle.PoolTransfer |
getStageInPoolTransfer(java.util.Map<java.lang.String,Bundle.PoolTransfer> stageInMap,
java.lang.String site,
int num,
boolean localTransfer)
Returns the appropriate stagein pool transfer for a particular site.
|
Job |
getSyncJob(java.lang.String site)
Returns the current synch job for a site.
|
protected java.lang.String |
getSyncJobBetweenLevelsName(java.lang.String site,
int level)
Returns the name of the job that acts as a synchronization node in
between stage in jobs of different levels.
|
protected void |
initializeBundleValues()
Initializes the bundle value variables, that are responsible determining
the bundle values.
|
protected java.util.Map<java.lang.String,Bundle.PoolTransfer> |
resetStageInMap(java.util.Map<java.lang.String,Bundle.PoolTransfer> stageInMap,
Implementation implementation,
java.util.Map<java.lang.String,Job> transientSynchJobMap,
int jobType,
boolean createChildSyncJob,
boolean localTransfer)
Resets the stage in map and adds the stage in jobs for each site per level.
|
protected void |
resetStageInMaps()
Resets the stage in map and adds the stage in jobs for each site per level.
|
addStageOutXFERNodes, doneStageIn, getDefaultBundleValueFromProperties, getStageOutPoolTransfer, resetStageOutMap, resetStageOutMaps
addInterSiteTXNodes, addJob, addRelation, addRelation, addStageInXFERNodes, addStageOutXFERNodes, appendAttribute, constructFileKey, createRegistrationJob, getJobPriority, logRefinerAction
loadImplementations
addStageInXFERNodes, getWorkflow, getXMLProducer, isSiteThirdParty, logConfigMessages, refinerPreferenceForLocalTransferJobs, refinerPreferenceForTransferJobLocation, runTPTOnRemoteSite, runTransferRemotely
public static final java.lang.String DESCRIPTION
public static final java.lang.String DEFAULT_LOCAL_STAGE_IN_CLUSTER_FACTOR
public static final java.lang.String DEFAULT_REMOTE_STAGE_IN_CLUSTER_FACTOR
public static final java.lang.String DEFAULT_LOCAL_STAGE_OUT_CLUSTER_FACTOR
public static final java.lang.String DEFAULT_REMOTE_STAGE_OUT_CLUSTER_FACTOR
protected java.util.Map<java.lang.String,Bundle.PoolTransfer> mStageInLocalMapPerLevel
protected java.util.Map<java.lang.String,Bundle.PoolTransfer> mStageInRemoteMapPerLevel
private int mCurrentSILevel
private java.util.Map<java.lang.String,Job> mSyncJobMap
public Cluster(ADag dag, PegasusBag bag)
dag
- the workflow to which transfer nodes need to be added.bag
- the bag of initialization objectsprotected void initializeBundleValues()
initializeBundleValues
in class Bundle
public void addStageInXFERNodes(Job job, java.util.Collection<FileTransfer> files, java.util.Collection<FileTransfer> symlinkFiles)
addStageInXFERNodes
in class Bundle
job
- Job
object corresponding to the node to
which the files are to be transferred to.files
- Collection of FileTransfer
objects containing the
information about source and destURL's.symlinkFiles
- Collection of FileTransfer
objects containing
source and destination file url's for symbolic linking
on compute site.public void addStageInXFERNodes(Job job, boolean localTransfer, java.util.Collection files, int jobType, java.util.Map<java.lang.String,Bundle.PoolTransfer> stageInMap, Bundle.BundleValue cValue, Implementation implementation)
addStageInXFERNodes
in class Bundle
job
- Job
object corresponding to the node to
which the files are to be transferred to.files
- Collection of FileTransfer
objects containing the
information about source and destURL's.jobType
- the type of transfer job being createdstageInMap
- Map indexed by site name that gives all the transfers for
that site.cValue
- used to determine the bundling factor to employ for a job.implementation
- the transfer implementation to use.localTransfer
- boolean indicating whether transfer has to happen on local site.protected void resetStageInMaps()
protected java.util.Map<java.lang.String,Bundle.PoolTransfer> resetStageInMap(java.util.Map<java.lang.String,Bundle.PoolTransfer> stageInMap, Implementation implementation, java.util.Map<java.lang.String,Job> transientSynchJobMap, int jobType, boolean createChildSyncJob, boolean localTransfer)
stageInMap
- the stagein map per level to be resetimplementation
- the transfer implementation corresponding to the map
being resettransientSynchJobMap
- transient map to keep track of synch jobs per site.jobType
- the type of Transfer Job createdcreateChildSyncJob
- boolean indicating whether to create a child synch job or not.protected java.lang.String getComputeJobBundleValue(Job job)
getComputeJobBundleValue
in class Bundle
job
- protected Bundle.PoolTransfer getStageInPoolTransfer(java.util.Map<java.lang.String,Bundle.PoolTransfer> stageInMap, java.lang.String site, int num, boolean localTransfer)
stageInMap
- map that indexes site to PoolTransfer objectssite
- the site for which the PT is reqd.num
- the number of stage in jobs required for that Pool.localTransfer
- whether the transfer needs to run on local site or not.public void done()
public java.lang.String getDescription()
getDescription
in class Bundle
protected int getSISiteBundleValue(java.lang.String site, java.lang.String deflt)
site
- the site at which the value is desired.deflt
- the default value.DEFAULT_LOCAL_STAGE_IN_CLUSTER_FACTOR
protected java.lang.String getSyncJobBetweenLevelsName(java.lang.String site, int level)
site
- the site of the transfer job.level
- the level of the jobprivate Job createSyncJobBetweenLevels(java.lang.String name)
name
- of the jobprotected void constructCondorKey(Job job, java.lang.String key, java.lang.String value)
job
- contains the job description.key
- the key of the profile.value
- the associated value.public Job getSyncJob(java.lang.String site)
site
-