Discussion:
DRMAA output redirection oddness
(too old to reply)
epistemyscott
2010-12-07 11:51:00 UTC
Permalink
Raw Message
Dear all,
I'm trying to use the Java DRMAA bindings to run jobs, and I'm running into problems when trying to redirect output.

For instance, when using the code below results in the following message in the qmaster messages file:

12/07/2010 11:19:28|worker|host123|W|job 71.1 failed on host host123.epistemy.com general opening input/output file because: 12/07/2010 11:19:27 [1024:14650]: error: can't open output file "$HOME/SGE-test/./SGE-test.log": No such file or directory

Touching the output files beforehand makes no difference.

If I comment out the setOutputPath and setErrorPath lines, I get the following:

12/07/2010 11:32:58|worker|host123|W|job 72.1 failed on host host123.epistemy.com general opening input/output file because: 12/07/2010 11:32:57 [1024:14705]: error: can't open output file "$HOME/SGE-test": No such file or directory

If I comment out the setWorkingDirectory line as well, it runs fine, but, of course, it puts all its output into my home directory with the wrong file names.

Should I be concerned at the apparent failure to expand $HOME, or is that just for logging purposes? Why does the second version seem to be trying to write its output directly onto the working directory? Does anyone have any suggestions on how to debug this? I'm running this using Java 1.6.0_17 on CentOS 5.5 and SGE 6.2u5.

Your help would be most appreciated.

Regards,
Scott

public static void submitJob(){
File runFolder = new File(System.getProperty("user.home"), "SGE-test");
runFolder.mkdir();

System.out.println("Folder created");

SessionFactory factory = SessionFactory.getFactory();
Session session = factory.getSession();

try{
session.init("");
JobTemplate job = session.createJobTemplate();

job.setRemoteCommand("pwd");

job.setJobName("SGE-test");
job.setBlockEmail(true);
job.setWorkingDirectory(JobTemplate.HOME_DIRECTORY+"SGE-test");
System.out.println("Working directory is: " + job.getWorkingDirectory());
job.setOutputPath(":"+JobTemplate.WORKING_DIRECTORY+"SGE-test.log");
job.setErrorPath(":"+JobTemplate.WORKING_DIRECTORY+"SGE-test.err.log");

String identifier = session.runJob(job);

System.out.println("Job submitted with identifier '" + identifier + "'.");

session.deleteJobTemplate(job);
}
catch(DrmaaException e){
System.out.println(e);
e.printStackTrace();
throw e;
}
finally{
try {
session.exit();
}
catch(DrmaaException e) {
System.out.println("Closing DRMAA session failed. " + e);
e.printStackTrace();
}
}
}


Scott Wilson
Developer
Epistemy
Energy Academy
Riccarton
EH14 4AS

tel: +44 131 564 0232
***@Epistemy.com
http://www.Epistemy.com/
Epistemy Limited is a company registered in Scotland, number SC365481.
Registered office: Epistemy Limited c/o Technology and Research Services, Heriot-Watt University, Riccarton, Edinburgh, EH14 4AS

------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302730

To unsubscribe from this discussion, e-mail: [users-***@gridengine.sunsource.net].
reuti
2010-12-07 13:03:27 UTC
Permalink
Raw Message
When you're using DRMAA, there's no shell to expand $HOME for you.
It's being taken literally, e.g. \$HOME. There are two ways to
reference the user's home directory from the workingDirectory,
inputPath, and outputPath properties. The first is just to use standard
UNIX conventions, e.g. ~. The second is to use the
http://gridengine.sunsource.net/nonav/source/browse/~checkout~/gridengine/doc/javadocs/org/ggf/drmaa/JobTemplate.html#HOME_DIRECTORY
like s.setWorkingDirectory(JobTemplate.HOME_DIRECTORY + "/SGE-test")
I thought this is what he is trying to do in the code he posted, but instead it's resolved to $HOME.

-- Reuti
Daniel
Post by epistemyscott
Dear all,
I'm trying to use the Java DRMAA bindings to run jobs, and I'm running into problems when trying to redirect output.
12/07/2010 11:19:28|worker|host123|W|job 71.1 failed on host host123.epistemy.com general opening input/output file because: 12/07/2010 11:19:27 [1024:14650]: error: can't open output file "$HOME/SGE-test/./SGE-test.log": No such file or directory
Touching the output files beforehand makes no difference.
12/07/2010 11:32:58|worker|host123|W|job 72.1 failed on host host123.epistemy.com general opening input/output file because: 12/07/2010 11:32:57 [1024:14705]: error: can't open output file "$HOME/SGE-test": No such file or directory
If I comment out the setWorkingDirectory line as well, it runs fine, but, of course, it puts all its output into my home directory with the wrong file names.
Should I be concerned at the apparent failure to expand $HOME, or is that just for logging purposes? Why does the second version seem to be trying to write its output directly onto the working directory? Does anyone have any suggestions on how to debug this? I'm running this using Java 1.6.0_17 on CentOS 5.5 and SGE 6.2u5.
Your help would be most appreciated.
Regards,
Scott
public static void submitJob(){
File runFolder = new File(System.getProperty("user.home"), "SGE-test");
runFolder.mkdir();
System.out.println("Folder created");
SessionFactory factory = SessionFactory.getFactory();
Session session = factory.getSession();
try{
session.init("");
JobTemplate job = session.createJobTemplate();
job.setRemoteCommand("pwd");
job.setJobName("SGE-test");
job.setBlockEmail(true);
job.setWorkingDirectory(JobTemplate.HOME_DIRECTORY+"SGE-test");
System.out.println("Working directory is: " + job.getWorkingDirectory());
job.setOutputPath(":"+JobTemplate.WORKING_DIRECTORY+"SGE-test.log");
job.setErrorPath(":"+JobTemplate.WORKING_DIRECTORY+"SGE-test.err.log");
String identifier = session.runJob(job);
System.out.println("Job submitted with identifier '" + identifier + "'.");
session.deleteJobTemplate(job);
}
catch(DrmaaException e){
System.out.println(e);
e.printStackTrace();
throw e;
}
finally{
try {
session.exit();
}
catch(DrmaaException e) {
System.out.println("Closing DRMAA session failed. " + e);
e.printStackTrace();
}
}
}
Scott Wilson
Developer
Epistemy
Energy Academy
Riccarton
EH14 4AS
tel: +44 131 564 0232
http://www.Epistemy.com/
Epistemy Limited is a company registered in Scotland, number SC365481.
Registered office: Epistemy Limited c/o Technology and Research Services, Heriot-Watt University, Riccarton, Edinburgh, EH14 4AS
------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302730
------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302763
------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302766

To unsubscribe from this discussion, e-mail: [users-***@gridengine.sunsource.net].
templedf
2010-12-07 13:07:43 UTC
Permalink
Raw Message
Sorry, missed that. I'm answering emails from a hotel room while
listening to a conference call. ;)

In that case, yes that should concern you. For now, try just using ~
instead of the placeholder. I'll try to reproduce it and file an issue
if needed.

Daniel
I am using the JobTemplate.HOME_DIRECTORY placeholder. See code at the end of initial post. The $HOME is coming from somewhere in Grid Engine.
Scott
Scott Wilson
Epistemy
http://gridengine.sunsource.net/nonav/source/browse/~checkout~/gridengine/doc/javadocs/org/ggf/drmaa/JobTemplate.html#HOME_DIRECTORY
like s.setWorkingDirectory(JobTemplate.HOME_DIRECTORY + "/SGE-test")
Daniel
Post by epistemyscott
Dear all,
I'm trying to use the Java DRMAA bindings to run jobs, and I'm running into problems when trying to redirect output.
12/07/2010 11:19:28|worker|host123|W|job 71.1 failed on host host123.epistemy.com general opening input/output file because: 12/07/2010 11:19:27 [1024:14650]: error: can't open output file "$HOME/SGE-test/./SGE-test.log": No such file or directory
Touching the output files beforehand makes no difference.
12/07/2010 11:32:58|worker|host123|W|job 72.1 failed on host host123.epistemy.com general opening input/output file because: 12/07/2010 11:32:57 [1024:14705]: error: can't open output file "$HOME/SGE-test": No such file or directory
If I comment out the setWorkingDirectory line as well, it runs fine, but, of course, it puts all its output into my home directory with the wrong file names.
Should I be concerned at the apparent failure to expand $HOME, or is that just for logging purposes? Why does the second version seem to be trying to write its output directly onto the working directory? Does anyone have any suggestions on how to debug this? I'm running this using Java 1.6.0_17 on CentOS 5.5 and SGE 6.2u5.
Your help would be most appreciated.
Regards,
Scott
public static void submitJob(){
File runFolder = new File(System.getProperty("user.home"), "SGE-test");
runFolder.mkdir();
System.out.println("Folder created");
SessionFactory factory = SessionFactory.getFactory();
Session session = factory.getSession();
try{
session.init("");
JobTemplate job = session.createJobTemplate();
job.setRemoteCommand("pwd");
job.setJobName("SGE-test");
job.setBlockEmail(true);
job.setWorkingDirectory(JobTemplate.HOME_DIRECTORY+"SGE-test");
System.out.println("Working directory is: " + job.getWorkingDirectory());
job.setOutputPath(":"+JobTemplate.WORKING_DIRECTORY+"SGE-test.log");
job.setErrorPath(":"+JobTemplate.WORKING_DIRECTORY+"SGE-test.err.log");
String identifier = session.runJob(job);
System.out.println("Job submitted with identifier '" + identifier + "'.");
session.deleteJobTemplate(job);
}
catch(DrmaaException e){
System.out.println(e);
e.printStackTrace();
throw e;
}
finally{
try {
session.exit();
}
catch(DrmaaException e) {
System.out.println("Closing DRMAA session failed. " + e);
e.printStackTrace();
}
}
}
Scott Wilson
Developer
Epistemy
Energy Academy
Riccarton
EH14 4AS
tel: +44 131 564 0232
http://www.Epistemy.com/
Epistemy Limited is a company registered in Scotland, number SC365481.
Registered office: Epistemy Limited c/o Technology and Research Services, Heriot-Watt University, Riccarton, Edinburgh, EH14 4AS
------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302730
------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302769

To unsubscribe from this discussion, e-mail: [users-***@gridengine.sunsource.net].
templedf
2010-12-07 13:25:08 UTC
Permalink
Raw Message
Well, then. Last suggestion: don't use anything. The default
assumption is that your starting point for all paths is the user's home
directory. If you set the workingDirectory to ./SGE-test, it should
resolve to ~/SGE-test.

Daniel
No bother. If only I got a penny every time *I* missed something. ;-)
12/07/2010 13:15:28|worker|host123|W|job 77.1 failed on host host123.epistemy.com general changing into working directory because: 12/07/2010 13:15:27 [1024:11046]: error: can't chdir to /SGE-test: No such file or directory
Is expansion of '~' not a shell feature, too?
Scott
Scott Wilson
Epistemy
Sorry, missed that. I'm answering emails from a hotel room while listening to a conference call. ;)
In that case, yes that should concern you. For now, try just using ~ instead of the placeholder. I'll try to reproduce it and file an issue if needed.
Daniel
I am using the JobTemplate.HOME_DIRECTORY placeholder. See code at the end of initial post. The $HOME is coming from somewhere in Grid Engine.
Scott
Scott Wilson
Epistemy
http://gridengine.sunsource.net/nonav/source/browse/~checkout~/gridengine/doc/javadocs/org/ggf/drmaa/JobTemplate.html#HOME_DIRECTORY
like s.setWorkingDirectory(JobTemplate.HOME_DIRECTORY + "/SGE-test")
Daniel
Post by epistemyscott
Dear all,
I'm trying to use the Java DRMAA bindings to run jobs, and I'm running into problems when trying to redirect output.
12/07/2010 11:19:28|worker|host123|W|job 71.1 failed on host host123.epistemy.com general opening input/output file because: 12/07/2010 11:19:27 [1024:14650]: error: can't open output file "$HOME/SGE-test/./SGE-test.log": No such file or directory
Touching the output files beforehand makes no difference.
12/07/2010 11:32:58|worker|host123|W|job 72.1 failed on host host123.epistemy.com general opening input/output file because: 12/07/2010 11:32:57 [1024:14705]: error: can't open output file "$HOME/SGE-test": No such file or directory
If I comment out the setWorkingDirectory line as well, it runs fine, but, of course, it puts all its output into my home directory with the wrong file names.
Should I be concerned at the apparent failure to expand $HOME, or is that just for logging purposes? Why does the second version seem to be trying to write its output directly onto the working directory? Does anyone have any suggestions on how to debug this? I'm running this using Java 1.6.0_17 on CentOS 5.5 and SGE 6.2u5.
Your help would be most appreciated.
Regards,
Scott
public static void submitJob(){
File runFolder = new File(System.getProperty("user.home"), "SGE-test");
runFolder.mkdir();
System.out.println("Folder created");
SessionFactory factory = SessionFactory.getFactory();
Session session = factory.getSession();
try{
session.init("");
JobTemplate job = session.createJobTemplate();
job.setRemoteCommand("pwd");
job.setJobName("SGE-test");
job.setBlockEmail(true);
job.setWorkingDirectory(JobTemplate.HOME_DIRECTORY+"SGE-test");
System.out.println("Working directory is: " + job.getWorkingDirectory());
job.setOutputPath(":"+JobTemplate.WORKING_DIRECTORY+"SGE-test.log");
job.setErrorPath(":"+JobTemplate.WORKING_DIRECTORY+"SGE-test.err.log");
String identifier = session.runJob(job);
System.out.println("Job submitted with identifier '" + identifier + "'.");
session.deleteJobTemplate(job);
}
catch(DrmaaException e){
System.out.println(e);
e.printStackTrace();
throw e;
}
finally{
try {
session.exit();
}
catch(DrmaaException e) {
System.out.println("Closing DRMAA session failed. " + e);
e.printStackTrace();
}
}
}
Scott Wilson
Developer
Epistemy
Energy Academy
Riccarton
EH14 4AS
tel: +44 131 564 0232
http://www.Epistemy.com/
Epistemy Limited is a company registered in Scotland, number SC365481.
Registered office: Epistemy Limited c/o Technology and Research Services, Heriot-Watt University, Riccarton, Edinburgh, EH14 4AS
------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302730
------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302773

To unsubscribe from this discussion, e-mail: [users-***@gridengine.sunsource.net].
reuti
2010-12-07 13:40:36 UTC
Permalink
Raw Message
Post by templedf
Well, then. Last suggestion: don't use anything. The default
assumption is that your starting point for all paths is the user's home
directory. If you set the workingDirectory to ./SGE-test, it should
resolve to ~/SGE-test.
In libs/japi/drmaa.c also $TASK_ID is referrenced. Hence the idea is not to use the environment variables of any shell, but the pseudo variables which can be used for -e/-o. Question is, why they are not expanded lateron.

What is `qstat -j <job_id>` showing for such a job of yours? For a `qsub -o ...`:

$ qstat -j 123
...
stdout_path_list: NONE:NONE:$HOME/test

should be there.

-- Reuti
Post by templedf
Daniel
No bother. If only I got a penny every time *I* missed something. ;-)
12/07/2010 13:15:28|worker|host123|W|job 77.1 failed on host host123.epistemy.com general changing into working directory because: 12/07/2010 13:15:27 [1024:11046]: error: can't chdir to /SGE-test: No such file or directory
Is expansion of '~' not a shell feature, too?
Scott
Scott Wilson
Epistemy
Sorry, missed that. I'm answering emails from a hotel room while listening to a conference call. ;)
In that case, yes that should concern you. For now, try just using ~ instead of the placeholder. I'll try to reproduce it and file an issue if needed.
Daniel
I am using the JobTemplate.HOME_DIRECTORY placeholder. See code at the end of initial post. The $HOME is coming from somewhere in Grid Engine.
Scott
Scott Wilson
Epistemy
http://gridengine.sunsource.net/nonav/source/browse/~checkout~/gridengine/doc/javadocs/org/ggf/drmaa/JobTemplate.html#HOME_DIRECTORY
like s.setWorkingDirectory(JobTemplate.HOME_DIRECTORY + "/SGE-test")
Daniel
Post by epistemyscott
Dear all,
I'm trying to use the Java DRMAA bindings to run jobs, and I'm running into problems when trying to redirect output.
12/07/2010 11:19:28|worker|host123|W|job 71.1 failed on host host123.epistemy.com general opening input/output file because: 12/07/2010 11:19:27 [1024:14650]: error: can't open output file "$HOME/SGE-test/./SGE-test.log": No such file or directory
Touching the output files beforehand makes no difference.
12/07/2010 11:32:58|worker|host123|W|job 72.1 failed on host host123.epistemy.com general opening input/output file because: 12/07/2010 11:32:57 [1024:14705]: error: can't open output file "$HOME/SGE-test": No such file or directory
If I comment out the setWorkingDirectory line as well, it runs fine, but, of course, it puts all its output into my home directory with the wrong file names.
Should I be concerned at the apparent failure to expand $HOME, or is that just for logging purposes? Why does the second version seem to be trying to write its output directly onto the working directory? Does anyone have any suggestions on how to debug this? I'm running this using Java 1.6.0_17 on CentOS 5.5 and SGE 6.2u5.
Your help would be most appreciated.
Regards,
Scott
public static void submitJob(){
File runFolder = new File(System.getProperty("user.home"), "SGE-test");
runFolder.mkdir();
System.out.println("Folder created");
SessionFactory factory = SessionFactory.getFactory();
Session session = factory.getSession();
try{
session.init("");
JobTemplate job = session.createJobTemplate();
job.setRemoteCommand("pwd");
job.setJobName("SGE-test");
job.setBlockEmail(true);
job.setWorkingDirectory(JobTemplate.HOME_DIRECTORY+"SGE-test");
System.out.println("Working directory is: " + job.getWorkingDirectory());
job.setOutputPath(":"+JobTemplate.WORKING_DIRECTORY+"SGE-test.log");
job.setErrorPath(":"+JobTemplate.WORKING_DIRECTORY+"SGE-test.err.log");
String identifier = session.runJob(job);
System.out.println("Job submitted with identifier '" + identifier + "'.");
session.deleteJobTemplate(job);
}
catch(DrmaaException e){
System.out.println(e);
e.printStackTrace();
throw e;
}
finally{
try {
session.exit();
}
catch(DrmaaException e) {
System.out.println("Closing DRMAA session failed. " + e);
e.printStackTrace();
}
}
}
Scott Wilson
Developer
Epistemy
Energy Academy
Riccarton
EH14 4AS
tel: +44 131 564 0232
http://www.Epistemy.com/
Epistemy Limited is a company registered in Scotland, number SC365481.
Registered office: Epistemy Limited c/o Technology and Research Services, Heriot-Watt University, Riccarton, Edinburgh, EH14 4AS
------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302730
------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302773
------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302778

To unsubscribe from this discussion, e-mail: [users-***@gridengine.sunsource.net].
epistemyscott
2010-12-07 14:29:05 UTC
Permalink
Raw Message
The DRMAA bindings get upset if the output path doesn't start with a ':'. The official format in the Javadoc is "[hostname]:file_path", although I'm not specifying any hostname - I presume this means "leave things as they are", though I may be wrong - I couldn't find any explicit explanation of what happens if you don't specify a host name.

I've tried setting the transferFiles attribute, which is mentioned in the Javadoc, to "new FileTransferMode(false, false, false)", but that results in an IllegalArgumentException. Also, the working directory can't be changed even if I don't specify an output or error path, and the working directory has no colons in it. Perhaps you've hit on the problem for the output and error paths, but I doubt it's the problem with the working directory.

Scott

Scott Wilson
Epistemy
stdout_path_list: NONE:host123:./SGE-test.log
This is interesting. I was always wondering about the two NONE in front of the output file name. The first one is set as I found if you specify a hostname before the output filename resp. output directory. I never saw the submit host popping up there.
Can it be, that it's coincidence because you prefix the output path by a ":" in your job.setOutputPath (which I wondered about already)?
-- Reuti
where host123 is the name (but not FQDN) of the host the job was submitted on. Could this be the issue? My home folder is on an NFS mount, so it should be accessible "locally" from all nodes. Could GE be trying to do file transfers? I haven't consciously set any up.
Scott
Scott Wilson
Epistemy
Post by reuti
Post by templedf
Well, then. Last suggestion: don't use anything. The default
assumption is that your starting point for all paths is the user's home
directory. If you set the workingDirectory to ./SGE-test, it should
resolve to ~/SGE-test.
In libs/japi/drmaa.c also $TASK_ID is referrenced. Hence the idea is not to use the environment variables of any shell, but the pseudo variables which can be used for -e/-o. Question is, why they are not expanded lateron.
$ qstat -j 123
...
stdout_path_list: NONE:NONE:$HOME/test
should be there.
-- Reuti
Post by templedf
Daniel
No bother. If only I got a penny every time *I* missed something. ;-)
12/07/2010 13:15:28|worker|host123|W|job 77.1 failed on host host123.epistemy.com general changing into working directory because: 12/07/2010 13:15:27 [1024:11046]: error: can't chdir to /SGE-test: No such file or directory
Is expansion of '~' not a shell feature, too?
Scott
Scott Wilson
Epistemy
Sorry, missed that. I'm answering emails from a hotel room while listening to a conference call. ;)
In that case, yes that should concern you. For now, try just using ~ instead of the placeholder. I'll try to reproduce it and file an issue if needed.
Daniel
I am using the JobTemplate.HOME_DIRECTORY placeholder. See code at the end of initial post. The $HOME is coming from somewhere in Grid Engine.
Scott
Scott Wilson
Epistemy
http://gridengine.sunsource.net/nonav/source/browse/~checkout~/gridengine/doc/javadocs/org/ggf/drmaa/JobTemplate.html#HOME_DIRECTORY
like s.setWorkingDirectory(JobTemplate.HOME_DIRECTORY + "/SGE-test")
Daniel
Post by epistemyscott
Dear all,
I'm trying to use the Java DRMAA bindings to run jobs, and I'm running into problems when trying to redirect output.
12/07/2010 11:19:28|worker|host123|W|job 71.1 failed on host host123.epistemy.com general opening input/output file because: 12/07/2010 11:19:27 [1024:14650]: error: can't open output file "$HOME/SGE-test/./SGE-test.log": No such file or directory
Touching the output files beforehand makes no difference.
12/07/2010 11:32:58|worker|host123|W|job 72.1 failed on host host123.epistemy.com general opening input/output file because: 12/07/2010 11:32:57 [1024:14705]: error: can't open output file "$HOME/SGE-test": No such file or directory
If I comment out the setWorkingDirectory line as well, it runs fine, but, of course, it puts all its output into my home directory with the wrong file names.
Should I be concerned at the apparent failure to expand $HOME, or is that just for logging purposes? Why does the second version seem to be trying to write its output directly onto the working directory? Does anyone have any suggestions on how to debug this? I'm running this using Java 1.6.0_17 on CentOS 5.5 and SGE 6.2u5.
Your help would be most appreciated.
Regards,
Scott
public static void submitJob(){
File runFolder = new File(System.getProperty("user.home"), "SGE-test");
runFolder.mkdir();
System.out.println("Folder created");
SessionFactory factory = SessionFactory.getFactory();
Session session = factory.getSession();
try{
session.init("");
JobTemplate job = session.createJobTemplate();
job.setRemoteCommand("pwd");
job.setJobName("SGE-test");
job.setBlockEmail(true);
job.setWorkingDirectory(JobTemplate.HOME_DIRECTORY+"SGE-test");
System.out.println("Working directory is: " + job.getWorkingDirectory());
job.setOutputPath(":"+JobTemplate.WORKING_DIRECTORY+"SGE-test.log");
job.setErrorPath(":"+JobTemplate.WORKING_DIRECTORY+"SGE-test.err.log");
String identifier = session.runJob(job);
System.out.println("Job submitted with identifier '" + identifier + "'.");
session.deleteJobTemplate(job);
}
catch(DrmaaException e){
System.out.println(e);
e.printStackTrace();
throw e;
}
finally{
try {
session.exit();
}
catch(DrmaaException e) {
System.out.println("Closing DRMAA session failed. " + e);
e.printStackTrace();
}
}
}
Scott Wilson
Developer
Epistemy
Energy Academy
Riccarton
EH14 4AS
tel: +44 131 564 0232
http://www.Epistemy.com/
Epistemy Limited is a company registered in Scotland, number SC365481.
Registered office: Epistemy Limited c/o Technology and Research Services, Heriot-Watt University, Riccarton, Edinburgh, EH14 4AS
------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302730
------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302773
------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302778
------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302789

To unsubscribe from this discussion, e-mail: [users-***@gridengine.sunsource.net].
reuti
2010-12-07 19:19:07 UTC
Permalink
Raw Message
Post by epistemyscott
The DRMAA bindings get upset if the output path doesn't start with a ':'. The official format in the Javadoc is "[hostname]:file_path", although I'm not specifying any hostname - I presume this means "leave things as they are", though I may be wrong - I couldn't find any explicit explanation of what happens if you don't specify a host name.
What about using a native specification and -o/-e/-wd for now, are the pseudo variables working there?
Post by epistemyscott
I've tried setting the transferFiles attribute, which is mentioned in the Javadoc, to "new FileTransferMode(false, false, false)", but that results in an IllegalArgumentException. Also, the working directory can't be changed even if I don't specify an output or error path, and the working directory has no colons in it. Perhaps you've hit on the problem for the output and error paths, but I doubt it's the problem with the working directory.
http://gridengine.sunsource.net/howto/filestaging/filestaging6.html

You defined "delegated_file_staging true"?

-- Reuti
Post by epistemyscott
Scott
Scott Wilson
Epistemy
stdout_path_list: NONE:host123:./SGE-test.log
This is interesting. I was always wondering about the two NONE in front of the output file name. The first one is set as I found if you specify a hostname before the output filename resp. output directory. I never saw the submit host popping up there.
Can it be, that it's coincidence because you prefix the output path by a ":" in your job.setOutputPath (which I wondered about already)?
-- Reuti
where host123 is the name (but not FQDN) of the host the job was submitted on. Could this be the issue? My home folder is on an NFS mount, so it should be accessible "locally" from all nodes. Could GE be trying to do file transfers? I haven't consciously set any up.
Scott
Scott Wilson
Epistemy
Post by reuti
Post by templedf
Well, then. Last suggestion: don't use anything. The default
assumption is that your starting point for all paths is the user's home
directory. If you set the workingDirectory to ./SGE-test, it should
resolve to ~/SGE-test.
In libs/japi/drmaa.c also $TASK_ID is referrenced. Hence the idea is not to use the environment variables of any shell, but the pseudo variables which can be used for -e/-o. Question is, why they are not expanded lateron.
$ qstat -j 123
...
stdout_path_list: NONE:NONE:$HOME/test
should be there.
-- Reuti
Post by templedf
Daniel
No bother. If only I got a penny every time *I* missed something. ;-)
12/07/2010 13:15:28|worker|host123|W|job 77.1 failed on host host123.epistemy.com general changing into working directory because: 12/07/2010 13:15:27 [1024:11046]: error: can't chdir to /SGE-test: No such file or directory
Is expansion of '~' not a shell feature, too?
Scott
Scott Wilson
Epistemy
Sorry, missed that. I'm answering emails from a hotel room while listening to a conference call. ;)
In that case, yes that should concern you. For now, try just using ~ instead of the placeholder. I'll try to reproduce it and file an issue if needed.
Daniel
I am using the JobTemplate.HOME_DIRECTORY placeholder. See code at the end of initial post. The $HOME is coming from somewhere in Grid Engine.
Scott
Scott Wilson
Epistemy
http://gridengine.sunsource.net/nonav/source/browse/~checkout~/gridengine/doc/javadocs/org/ggf/drmaa/JobTemplate.html#HOME_DIRECTORY
like s.setWorkingDirectory(JobTemplate.HOME_DIRECTORY + "/SGE-test")
Daniel
Post by epistemyscott
Dear all,
I'm trying to use the Java DRMAA bindings to run jobs, and I'm running into problems when trying to redirect output.
12/07/2010 11:19:28|worker|host123|W|job 71.1 failed on host host123.epistemy.com general opening input/output file because: 12/07/2010 11:19:27 [1024:14650]: error: can't open output file "$HOME/SGE-test/./SGE-test.log": No such file or directory
Touching the output files beforehand makes no difference.
12/07/2010 11:32:58|worker|host123|W|job 72.1 failed on host host123.epistemy.com general opening input/output file because: 12/07/2010 11:32:57 [1024:14705]: error: can't open output file "$HOME/SGE-test": No such file or directory
If I comment out the setWorkingDirectory line as well, it runs fine, but, of course, it puts all its output into my home directory with the wrong file names.
Should I be concerned at the apparent failure to expand $HOME, or is that just for logging purposes? Why does the second version seem to be trying to write its output directly onto the working directory? Does anyone have any suggestions on how to debug this? I'm running this using Java 1.6.0_17 on CentOS 5.5 and SGE 6.2u5.
Your help would be most appreciated.
Regards,
Scott
public static void submitJob(){
File runFolder = new File(System.getProperty("user.home"), "SGE-test");
runFolder.mkdir();
System.out.println("Folder created");
SessionFactory factory = SessionFactory.getFactory();
Session session = factory.getSession();
try{
session.init("");
JobTemplate job = session.createJobTemplate();
job.setRemoteCommand("pwd");
job.setJobName("SGE-test");
job.setBlockEmail(true);
job.setWorkingDirectory(JobTemplate.HOME_DIRECTORY+"SGE-test");
System.out.println("Working directory is: " + job.getWorkingDirectory());
job.setOutputPath(":"+JobTemplate.WORKING_DIRECTORY+"SGE-test.log");
job.setErrorPath(":"+JobTemplate.WORKING_DIRECTORY+"SGE-test.err.log");
String identifier = session.runJob(job);
System.out.println("Job submitted with identifier '" + identifier + "'.");
session.deleteJobTemplate(job);
}
catch(DrmaaException e){
System.out.println(e);
e.printStackTrace();
throw e;
}
finally{
try {
session.exit();
}
catch(DrmaaException e) {
System.out.println("Closing DRMAA session failed. " + e);
e.printStackTrace();
}
}
}
Scott Wilson
Developer
Epistemy
Energy Academy
Riccarton
EH14 4AS
tel: +44 131 564 0232
http://www.Epistemy.com/
Epistemy Limited is a company registered in Scotland, number SC365481.
Registered office: Epistemy Limited c/o Technology and Research Services, Heriot-Watt University, Riccarton, Edinburgh, EH14 4AS
------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302730
------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302773
------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302778
------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302789
------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302889

To unsubscribe from this discussion, e-mail: [users-***@gridengine.sunsource.net].
epistemyscott
2010-12-08 12:10:23 UTC
Permalink
Raw Message
Post by reuti
Post by epistemyscott
The DRMAA bindings get upset if the output path doesn't start with a ':'. The official format in the Javadoc is "[hostname]:file_path", although I'm not specifying any hostname - I presume this means "leave things as they are", though I may be wrong - I couldn't find any explicit explanation of what happens if you don't specify a host name.
What about using a native specification and -o/-e/-wd for now, are the pseudo variables working there?
The stuff I'm writing has to work with TORQUE as well (hence the DRMAA), so I'd like to stay away from native specs as much as possible. Nonetheless, I gave this a try, and it doesn't work - using job.setNativeSpecification("-wd ~/SGE-test -o SGE-test.log -e SGE-test.err.log"); and removing the DRMAA lines fails with the familiar message:

12/08/2010 11:54:28|worker|host123|W|job 92.1 failed on host host123.epistemy.com general changing into working directory because: 12/08/2010 11:54:27 [1024:14268]: error: can't chdir to /SGE-test: No such file or directory
Post by reuti
Post by epistemyscott
I've tried setting the transferFiles attribute, which is mentioned in the Javadoc, to "new FileTransferMode(false, false, false)", but that results in an IllegalArgumentException. Also, the working directory can't be changed even if I don't specify an output or error path, and the working directory has no colons in it. Perhaps you've hit on the problem for the output and error paths, but I doubt it's the problem with the working directory.
http://gridengine.sunsource.net/howto/filestaging/filestaging6.html
You defined "delegated_file_staging true"?
No. I don't want to do delegated file staging - I want to use a shared network file system. Although it doesn't work with DRMAA, doing "qsub -wd ~/SGE-test -o test.log -e test.err.log ~/testScript.sh" on the command line works fine.

Scott
Post by reuti
-- Reuti
Post by epistemyscott
Scott
Scott Wilson
Epistemy
stdout_path_list: NONE:host123:./SGE-test.log
This is interesting. I was always wondering about the two NONE in front of the output file name. The first one is set as I found if you specify a hostname before the output filename resp. output directory. I never saw the submit host popping up there.
Can it be, that it's coincidence because you prefix the output path by a ":" in your job.setOutputPath (which I wondered about already)?
-- Reuti
where host123 is the name (but not FQDN) of the host the job was submitted on. Could this be the issue? My home folder is on an NFS mount, so it should be accessible "locally" from all nodes. Could GE be trying to do file transfers? I haven't consciously set any up.
Scott
Scott Wilson
Epistemy
Post by reuti
Post by templedf
Well, then. Last suggestion: don't use anything. The default
assumption is that your starting point for all paths is the user's home
directory. If you set the workingDirectory to ./SGE-test, it should
resolve to ~/SGE-test.
In libs/japi/drmaa.c also $TASK_ID is referrenced. Hence the idea is not to use the environment variables of any shell, but the pseudo variables which can be used for -e/-o. Question is, why they are not expanded lateron.
$ qstat -j 123
...
stdout_path_list: NONE:NONE:$HOME/test
should be there.
-- Reuti
Post by templedf
Daniel
No bother. If only I got a penny every time *I* missed something. ;-)
12/07/2010 13:15:28|worker|host123|W|job 77.1 failed on host host123.epistemy.com general changing into working directory because: 12/07/2010 13:15:27 [1024:11046]: error: can't chdir to /SGE-test: No such file or directory
Is expansion of '~' not a shell feature, too?
Scott
Scott Wilson
Epistemy
Sorry, missed that. I'm answering emails from a hotel room while listening to a conference call. ;)
In that case, yes that should concern you. For now, try just using ~ instead of the placeholder. I'll try to reproduce it and file an issue if needed.
Daniel
I am using the JobTemplate.HOME_DIRECTORY placeholder. See code at the end of initial post. The $HOME is coming from somewhere in Grid Engine.
Scott
Scott Wilson
Epistemy
http://gridengine.sunsource.net/nonav/source/browse/~checkout~/gridengine/doc/javadocs/org/ggf/drmaa/JobTemplate.html#HOME_DIRECTORY
like s.setWorkingDirectory(JobTemplate.HOME_DIRECTORY + "/SGE-test")
Daniel
Post by epistemyscott
Dear all,
I'm trying to use the Java DRMAA bindings to run jobs, and I'm running into problems when trying to redirect output.
12/07/2010 11:19:28|worker|host123|W|job 71.1 failed on host host123.epistemy.com general opening input/output file because: 12/07/2010 11:19:27 [1024:14650]: error: can't open output file "$HOME/SGE-test/./SGE-test.log": No such file or directory
Touching the output files beforehand makes no difference.
12/07/2010 11:32:58|worker|host123|W|job 72.1 failed on host host123.epistemy.com general opening input/output file because: 12/07/2010 11:32:57 [1024:14705]: error: can't open output file "$HOME/SGE-test": No such file or directory
If I comment out the setWorkingDirectory line as well, it runs fine, but, of course, it puts all its output into my home directory with the wrong file names.
Should I be concerned at the apparent failure to expand $HOME, or is that just for logging purposes? Why does the second version seem to be trying to write its output directly onto the working directory? Does anyone have any suggestions on how to debug this? I'm running this using Java 1.6.0_17 on CentOS 5.5 and SGE 6.2u5.
Your help would be most appreciated.
Regards,
Scott
public static void submitJob(){
File runFolder = new File(System.getProperty("user.home"), "SGE-test");
runFolder.mkdir();
System.out.println("Folder created");
SessionFactory factory = SessionFactory.getFactory();
Session session = factory.getSession();
try{
session.init("");
JobTemplate job = session.createJobTemplate();
job.setRemoteCommand("pwd");
job.setJobName("SGE-test");
job.setBlockEmail(true);
job.setWorkingDirectory(JobTemplate.HOME_DIRECTORY+"SGE-test");
System.out.println("Working directory is: " + job.getWorkingDirectory());
job.setOutputPath(":"+JobTemplate.WORKING_DIRECTORY+"SGE-test.log");
job.setErrorPath(":"+JobTemplate.WORKING_DIRECTORY+"SGE-test.err.log");
String identifier = session.runJob(job);
System.out.println("Job submitted with identifier '" + identifier + "'.");
session.deleteJobTemplate(job);
}
catch(DrmaaException e){
System.out.println(e);
e.printStackTrace();
throw e;
}
finally{
try {
session.exit();
}
catch(DrmaaException e) {
System.out.println("Closing DRMAA session failed. " + e);
e.printStackTrace();
}
}
}
Scott Wilson
Developer
Epistemy
Energy Academy
Riccarton
EH14 4AS
tel: +44 131 564 0232
http://www.Epistemy.com/
Epistemy Limited is a company registered in Scotland, number SC365481.
Registered office: Epistemy Limited c/o Technology and Research Services, Heriot-Watt University, Riccarton, Edinburgh, EH14 4AS
------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302730
------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302773
------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302778
------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302789
------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302889
------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=303097

To unsubscribe from this discussion, e-mail: [users-***@gridengine.sunsource.net].
reuti
2010-12-08 13:29:29 UTC
Permalink
Raw Message
Post by epistemyscott
Post by reuti
Post by epistemyscott
The DRMAA bindings get upset if the output path doesn't start with a ':'. The official format in the Javadoc is "[hostname]:file_path", although I'm not specifying any hostname - I presume this means "leave things as they are", though I may be wrong - I couldn't find any explicit explanation of what happens if you don't specify a host name.
What about using a native specification and -o/-e/-wd for now, are the pseudo variables working there?
AFAICS -wd doesn't support pseudo variables. So I would assume you need the full path to /home/whoever/SGE-test
Post by epistemyscott
12/08/2010 11:54:28|worker|host123|W|job 92.1 failed on host host123.epistemy.com general changing into working directory because: 12/08/2010 11:54:27 [1024:14268]: error: can't chdir to /SGE-test: No such file or directory
Post by reuti
Post by epistemyscott
I've tried setting the transferFiles attribute, which is mentioned in the Javadoc, to "new FileTransferMode(false, false, false)", but that results in an IllegalArgumentException. Also, the working directory can't be changed even if I don't specify an output or error path, and the working directory has no colons in it. Perhaps you've hit on the problem for the output and error paths, but I doubt it's the problem with the working directory.
http://gridengine.sunsource.net/howto/filestaging/filestaging6.html
You defined "delegated_file_staging true"?
No. I don't want to do delegated file staging - I want to use a shared network file system. Although it doesn't work with DRMAA, doing "qsub -wd ~/SGE-test -o test.log -e test.err.log ~/testScript.sh" on the command line works fine.
Yes, but here the shell will expand it. When you try:

$ qsub -wd "~/SGE-test" -o test.log -e test.err.log ~/testScript.sh

it should also fail. OTOH: -o "~/test.log" (not expanded by shell here!) should work, but it doesn't. It crashed the sge_execd for me in most cases. Same when you put it in the job script with:

#!/bin/sh
#$ -e ~/test1 -o ~/test2
...

Maybe the expansion of pseudo variables is broken per se as I see in the messages file:

12/08/2010 14:24:28| main|pc15370|E|invalid user name "/***@361350267310343^_^H356qܷyqܷ@361350267310343^_^H^P"
12/08/2010 14:24:28| main|pc15370|E|invalid user name "/***@361350267310343^_^H356qܷyqܷ@361350267310343^_^H^P"

We use job generators and they always put full pathnames there, so it never hit us.

When you observe the same, I would suggest to file an issue about it. Maybe the DRMAA problem is only showing up a different issue.

-- Reuti

------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=303152

To unsubscribe from this discussion, e-mail: [users-***@gridengine.sunsource.net].
reuti
2010-12-08 13:47:23 UTC
Permalink
Raw Message
Post by epistemyscott
Post by reuti
Post by epistemyscott
I've tried setting the transferFiles attribute, which is mentioned in the Javadoc, to "new FileTransferMode(false, false, false)", but that results in an IllegalArgumentException. Also, the working directory can't be changed even if I don't specify an output or error path, and the working directory has no colons in it. Perhaps you've hit on the problem for the output and error paths, but I doubt it's the problem with the working directory.
http://gridengine.sunsource.net/howto/filestaging/filestaging6.html
You defined "delegated_file_staging true"?
No. I don't want to do delegated file staging - I want to use a shared network file system.
Yes, but I think "FileTransferMode" is to set up the staging of input,output,error files. So you just want to be sure, that it's switched off?

-- Reuti

------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=303156

To unsubscribe from this discussion, e-mail: [users-***@gridengine.sunsource.net].
reuti
2010-12-10 22:04:24 UTC
Permalink
Raw Message
Post by epistemyscott
Post by reuti
Post by epistemyscott
The DRMAA bindings get upset if the output path doesn't start with a ':'. The official format in the Javadoc is "[hostname]:file_path", although I'm not specifying any hostname - I presume this means "leave things as they are", though I may be wrong - I couldn't find any explicit explanation of what happens if you don't specify a host name.
What about using a native specification and -o/-e/-wd for now, are the pseudo variables working there?
12/08/2010 11:54:28|worker|host123|W|job 92.1 failed on host host123.epistemy.com general changing into working directory because: 12/08/2010 11:54:27 [1024:14268]: error: can't chdir to /SGE-test: No such file or directory
What I see for now is, that "$drmaa_hd_ph$" is correctly replaced by "$HOME". But it seems, that the replacement takes place at the wrong time. I mean:

job.setErrorPath(":"+"$HOME/SGE-test.err.log");

is working fine, while your:

job.setOutputPath(":"+JobTemplate.WORKING_DIRECTORY+"SGE-test.log");

fails, as it's happening with a direct:

job.setOutputPath(":"+"$drmaa_hd_ph$SGE-test.log");

-- Reuti

------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=304075

To unsubscribe from this discussion, e-mail: [users-***@gridengine.sunsource.net].
reuti
2010-12-12 13:32:54 UTC
Permalink
Raw Message
Post by epistemyscott
Post by reuti
Post by epistemyscott
The DRMAA bindings get upset if the output path doesn't start with a ':'. The official format in the Javadoc is "[hostname]:file_path", although I'm not specifying any hostname - I presume this means "leave things as they are", though I may be wrong - I couldn't find any explicit explanation of what happens if you don't specify a host name.
What about using a native specification and -o/-e/-wd for now, are the pseudo variables working there?
12/08/2010 11:54:28|worker|host123|W|job 92.1 failed on host host123.epistemy.com general changing into working directory because: 12/08/2010 11:54:27 [1024:14268]: error: can't chdir to /SGE-test: No such file or directory
Did you also observer, that "job.setWorkingDirectory" will create the target directory, when it's not present?

Is this a feature or a bug? The plain `qsub -wd <directory>` will send the job into error state instead, which is what I expect. Though the DRMAA specification could be different.

-- Reuti

------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=304686

To unsubscribe from this discussion, e-mail: [users-***@gridengine.sunsource.net].
epistemyscott
2010-12-07 13:54:36 UTC
Permalink
Raw Message
I get:

stdout_path_list: NONE:host123:./SGE-test.log

where host123 is the name (but not FQDN) of the host the job was submitted on. Could this be the issue? My home folder is on an NFS mount, so it should be accessible "locally" from all nodes. Could GE be trying to do file transfers? I haven't consciously set any up.

Scott

Scott Wilson
Epistemy
Post by reuti
Post by templedf
Well, then. Last suggestion: don't use anything. The default
assumption is that your starting point for all paths is the user's home
directory. If you set the workingDirectory to ./SGE-test, it should
resolve to ~/SGE-test.
In libs/japi/drmaa.c also $TASK_ID is referrenced. Hence the idea is not to use the environment variables of any shell, but the pseudo variables which can be used for -e/-o. Question is, why they are not expanded lateron.
$ qstat -j 123
...
stdout_path_list: NONE:NONE:$HOME/test
should be there.
-- Reuti
Post by templedf
Daniel
No bother. If only I got a penny every time *I* missed something. ;-)
12/07/2010 13:15:28|worker|host123|W|job 77.1 failed on host host123.epistemy.com general changing into working directory because: 12/07/2010 13:15:27 [1024:11046]: error: can't chdir to /SGE-test: No such file or directory
Is expansion of '~' not a shell feature, too?
Scott
Scott Wilson
Epistemy
Sorry, missed that. I'm answering emails from a hotel room while listening to a conference call. ;)
In that case, yes that should concern you. For now, try just using ~ instead of the placeholder. I'll try to reproduce it and file an issue if needed.
Daniel
I am using the JobTemplate.HOME_DIRECTORY placeholder. See code at the end of initial post. The $HOME is coming from somewhere in Grid Engine.
Scott
Scott Wilson
Epistemy
http://gridengine.sunsource.net/nonav/source/browse/~checkout~/gridengine/doc/javadocs/org/ggf/drmaa/JobTemplate.html#HOME_DIRECTORY
like s.setWorkingDirectory(JobTemplate.HOME_DIRECTORY + "/SGE-test")
Daniel
Post by epistemyscott
Dear all,
I'm trying to use the Java DRMAA bindings to run jobs, and I'm running into problems when trying to redirect output.
12/07/2010 11:19:28|worker|host123|W|job 71.1 failed on host host123.epistemy.com general opening input/output file because: 12/07/2010 11:19:27 [1024:14650]: error: can't open output file "$HOME/SGE-test/./SGE-test.log": No such file or directory
Touching the output files beforehand makes no difference.
12/07/2010 11:32:58|worker|host123|W|job 72.1 failed on host host123.epistemy.com general opening input/output file because: 12/07/2010 11:32:57 [1024:14705]: error: can't open output file "$HOME/SGE-test": No such file or directory
If I comment out the setWorkingDirectory line as well, it runs fine, but, of course, it puts all its output into my home directory with the wrong file names.
Should I be concerned at the apparent failure to expand $HOME, or is that just for logging purposes? Why does the second version seem to be trying to write its output directly onto the working directory? Does anyone have any suggestions on how to debug this? I'm running this using Java 1.6.0_17 on CentOS 5.5 and SGE 6.2u5.
Your help would be most appreciated.
Regards,
Scott
public static void submitJob(){
File runFolder = new File(System.getProperty("user.home"), "SGE-test");
runFolder.mkdir();
System.out.println("Folder created");
SessionFactory factory = SessionFactory.getFactory();
Session session = factory.getSession();
try{
session.init("");
JobTemplate job = session.createJobTemplate();
job.setRemoteCommand("pwd");
job.setJobName("SGE-test");
job.setBlockEmail(true);
job.setWorkingDirectory(JobTemplate.HOME_DIRECTORY+"SGE-test");
System.out.println("Working directory is: " + job.getWorkingDirectory());
job.setOutputPath(":"+JobTemplate.WORKING_DIRECTORY+"SGE-test.log");
job.setErrorPath(":"+JobTemplate.WORKING_DIRECTORY+"SGE-test.err.log");
String identifier = session.runJob(job);
System.out.println("Job submitted with identifier '" + identifier + "'.");
session.deleteJobTemplate(job);
}
catch(DrmaaException e){
System.out.println(e);
e.printStackTrace();
throw e;
}
finally{
try {
session.exit();
}
catch(DrmaaException e) {
System.out.println("Closing DRMAA session failed. " + e);
e.printStackTrace();
}
}
}
Scott Wilson
Developer
Epistemy
Energy Academy
Riccarton
EH14 4AS
tel: +44 131 564 0232
http://www.Epistemy.com/
Epistemy Limited is a company registered in Scotland, number SC365481.
Registered office: Epistemy Limited c/o Technology and Research Services, Heriot-Watt University, Riccarton, Edinburgh, EH14 4AS
------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302730
------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302773
------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302778
------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302783

To unsubscribe from this discussion, e-mail: [users-***@gridengine.sunsource.net].
reuti
2010-12-07 15:40:59 UTC
Permalink
Raw Message
Please don't CC me. My answers may go to the wrong address.
stdout_path_list: NONE:host123:./SGE-test.log
This is interesting. I was always wondering about the two NONE in front of the output file name. The first one is set as I found if you specify a hostname before the output filename resp. output directory. I never saw the submit host popping up there.

Can it be, that it's coincidence because you prefix the output path by a ":" in your job.setOutputPath (which I wondered about already)?

-- Reuti
where host123 is the name (but not FQDN) of the host the job was submitted on. Could this be the issue? My home folder is on an NFS mount, so it should be accessible "locally" from all nodes. Could GE be trying to do file transfers? I haven't consciously set any up.
Scott
Scott Wilson
Epistemy
Post by reuti
Post by templedf
Well, then. Last suggestion: don't use anything. The default
assumption is that your starting point for all paths is the user's home
directory. If you set the workingDirectory to ./SGE-test, it should
resolve to ~/SGE-test.
In libs/japi/drmaa.c also $TASK_ID is referrenced. Hence the idea is not to use the environment variables of any shell, but the pseudo variables which can be used for -e/-o. Question is, why they are not expanded lateron.
$ qstat -j 123
...
stdout_path_list: NONE:NONE:$HOME/test
should be there.
-- Reuti
Post by templedf
Daniel
No bother. If only I got a penny every time *I* missed something. ;-)
12/07/2010 13:15:28|worker|host123|W|job 77.1 failed on host host123.epistemy.com general changing into working directory because: 12/07/2010 13:15:27 [1024:11046]: error: can't chdir to /SGE-test: No such file or directory
Is expansion of '~' not a shell feature, too?
Scott
Scott Wilson
Epistemy
Sorry, missed that. I'm answering emails from a hotel room while listening to a conference call. ;)
In that case, yes that should concern you. For now, try just using ~ instead of the placeholder. I'll try to reproduce it and file an issue if needed.
Daniel
I am using the JobTemplate.HOME_DIRECTORY placeholder. See code at the end of initial post. The $HOME is coming from somewhere in Grid Engine.
Scott
Scott Wilson
Epistemy
http://gridengine.sunsource.net/nonav/source/browse/~checkout~/gridengine/doc/javadocs/org/ggf/drmaa/JobTemplate.html#HOME_DIRECTORY
like s.setWorkingDirectory(JobTemplate.HOME_DIRECTORY + "/SGE-test")
Daniel
Post by epistemyscott
Dear all,
I'm trying to use the Java DRMAA bindings to run jobs, and I'm running into problems when trying to redirect output.
12/07/2010 11:19:28|worker|host123|W|job 71.1 failed on host host123.epistemy.com general opening input/output file because: 12/07/2010 11:19:27 [1024:14650]: error: can't open output file "$HOME/SGE-test/./SGE-test.log": No such file or directory
Touching the output files beforehand makes no difference.
12/07/2010 11:32:58|worker|host123|W|job 72.1 failed on host host123.epistemy.com general opening input/output file because: 12/07/2010 11:32:57 [1024:14705]: error: can't open output file "$HOME/SGE-test": No such file or directory
If I comment out the setWorkingDirectory line as well, it runs fine, but, of course, it puts all its output into my home directory with the wrong file names.
Should I be concerned at the apparent failure to expand $HOME, or is that just for logging purposes? Why does the second version seem to be trying to write its output directly onto the working directory? Does anyone have any suggestions on how to debug this? I'm running this using Java 1.6.0_17 on CentOS 5.5 and SGE 6.2u5.
Your help would be most appreciated.
Regards,
Scott
public static void submitJob(){
File runFolder = new File(System.getProperty("user.home"), "SGE-test");
runFolder.mkdir();
System.out.println("Folder created");
SessionFactory factory = SessionFactory.getFactory();
Session session = factory.getSession();
try{
session.init("");
JobTemplate job = session.createJobTemplate();
job.setRemoteCommand("pwd");
job.setJobName("SGE-test");
job.setBlockEmail(true);
job.setWorkingDirectory(JobTemplate.HOME_DIRECTORY+"SGE-test");
System.out.println("Working directory is: " + job.getWorkingDirectory());
job.setOutputPath(":"+JobTemplate.WORKING_DIRECTORY+"SGE-test.log");
job.setErrorPath(":"+JobTemplate.WORKING_DIRECTORY+"SGE-test.err.log");
String identifier = session.runJob(job);
System.out.println("Job submitted with identifier '" + identifier + "'.");
session.deleteJobTemplate(job);
}
catch(DrmaaException e){
System.out.println(e);
e.printStackTrace();
throw e;
}
finally{
try {
session.exit();
}
catch(DrmaaException e) {
System.out.println("Closing DRMAA session failed. " + e);
e.printStackTrace();
}
}
}
Scott Wilson
Developer
Epistemy
Energy Academy
Riccarton
EH14 4AS
tel: +44 131 564 0232
http://www.Epistemy.com/
Epistemy Limited is a company registered in Scotland, number SC365481.
Registered office: Epistemy Limited c/o Technology and Research Services, Heriot-Watt University, Riccarton, Edinburgh, EH14 4AS
------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302730
------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302773
------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302778
------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302807

To unsubscribe from this discussion, e-mail: [users-***@gridengine.sunsource.net].
epistemyscott
2010-12-07 15:51:51 UTC
Permalink
Raw Message
Post by reuti
Please don't CC me. My answers may go to the wrong address.
My apologies. I have much the same problem sometimes, which is probably why you were CC-ed in the first place.

Scott
Post by reuti
stdout_path_list: NONE:host123:./SGE-test.log
This is interesting. I was always wondering about the two NONE in front of the output file name. The first one is set as I found if you specify a hostname before the output filename resp. output directory. I never saw the submit host popping up there.
Can it be, that it's coincidence because you prefix the output path by a ":" in your job.setOutputPath (which I wondered about already)?
-- Reuti
where host123 is the name (but not FQDN) of the host the job was submitted on. Could this be the issue? My home folder is on an NFS mount, so it should be accessible "locally" from all nodes. Could GE be trying to do file transfers? I haven't consciously set any up.
Scott
Scott Wilson
Epistemy
Post by reuti
Post by templedf
Well, then. Last suggestion: don't use anything. The default
assumption is that your starting point for all paths is the user's home
directory. If you set the workingDirectory to ./SGE-test, it should
resolve to ~/SGE-test.
In libs/japi/drmaa.c also $TASK_ID is referrenced. Hence the idea is not to use the environment variables of any shell, but the pseudo variables which can be used for -e/-o. Question is, why they are not expanded lateron.
$ qstat -j 123
...
stdout_path_list: NONE:NONE:$HOME/test
should be there.
-- Reuti
Post by templedf
Daniel
No bother. If only I got a penny every time *I* missed something. ;-)
12/07/2010 13:15:28|worker|host123|W|job 77.1 failed on host host123.epistemy.com general changing into working directory because: 12/07/2010 13:15:27 [1024:11046]: error: can't chdir to /SGE-test: No such file or directory
Is expansion of '~' not a shell feature, too?
Scott
Scott Wilson
Epistemy
Sorry, missed that. I'm answering emails from a hotel room while listening to a conference call. ;)
In that case, yes that should concern you. For now, try just using ~ instead of the placeholder. I'll try to reproduce it and file an issue if needed.
Daniel
I am using the JobTemplate.HOME_DIRECTORY placeholder. See code at the end of initial post. The $HOME is coming from somewhere in Grid Engine.
Scott
Scott Wilson
Epistemy
http://gridengine.sunsource.net/nonav/source/browse/~checkout~/gridengine/doc/javadocs/org/ggf/drmaa/JobTemplate.html#HOME_DIRECTORY
like s.setWorkingDirectory(JobTemplate.HOME_DIRECTORY + "/SGE-test")
Daniel
Post by epistemyscott
Dear all,
I'm trying to use the Java DRMAA bindings to run jobs, and I'm running into problems when trying to redirect output.
12/07/2010 11:19:28|worker|host123|W|job 71.1 failed on host host123.epistemy.com general opening input/output file because: 12/07/2010 11:19:27 [1024:14650]: error: can't open output file "$HOME/SGE-test/./SGE-test.log": No such file or directory
Touching the output files beforehand makes no difference.
12/07/2010 11:32:58|worker|host123|W|job 72.1 failed on host host123.epistemy.com general opening input/output file because: 12/07/2010 11:32:57 [1024:14705]: error: can't open output file "$HOME/SGE-test": No such file or directory
If I comment out the setWorkingDirectory line as well, it runs fine, but, of course, it puts all its output into my home directory with the wrong file names.
Should I be concerned at the apparent failure to expand $HOME, or is that just for logging purposes? Why does the second version seem to be trying to write its output directly onto the working directory? Does anyone have any suggestions on how to debug this? I'm running this using Java 1.6.0_17 on CentOS 5.5 and SGE 6.2u5.
Your help would be most appreciated.
Regards,
Scott
public static void submitJob(){
File runFolder = new File(System.getProperty("user.home"), "SGE-test");
runFolder.mkdir();
System.out.println("Folder created");
SessionFactory factory = SessionFactory.getFactory();
Session session = factory.getSession();
try{
session.init("");
JobTemplate job = session.createJobTemplate();
job.setRemoteCommand("pwd");
job.setJobName("SGE-test");
job.setBlockEmail(true);
job.setWorkingDirectory(JobTemplate.HOME_DIRECTORY+"SGE-test");
System.out.println("Working directory is: " + job.getWorkingDirectory());
job.setOutputPath(":"+JobTemplate.WORKING_DIRECTORY+"SGE-test.log");
job.setErrorPath(":"+JobTemplate.WORKING_DIRECTORY+"SGE-test.err.log");
String identifier = session.runJob(job);
System.out.println("Job submitted with identifier '" + identifier + "'.");
session.deleteJobTemplate(job);
}
catch(DrmaaException e){
System.out.println(e);
e.printStackTrace();
throw e;
}
finally{
try {
session.exit();
}
catch(DrmaaException e) {
System.out.println("Closing DRMAA session failed. " + e);
e.printStackTrace();
}
}
}
Scott Wilson
Developer
Epistemy
Energy Academy
Riccarton
EH14 4AS
tel: +44 131 564 0232
http://www.Epistemy.com/
Epistemy Limited is a company registered in Scotland, number SC365481.
Registered office: Epistemy Limited c/o Technology and Research Services, Heriot-Watt University, Riccarton, Edinburgh, EH14 4AS
------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302730
------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302773
------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302778
------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302807
------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302812

To unsubscribe from this discussion, e-mail: [users-***@gridengine.sunsource.net].
epistemyscott
2010-12-07 13:40:21 UTC
Permalink
Raw Message
Grid Engine really doesn't like my directories:

12/07/2010 13:28:43|worker|host123|W|job 78.1 failed on host host123.epistemy.com general changing into working directory because: 12/07/2010 13:28:42 [1024:11068]: error: can't chdir to ./SGE-test: No such file or directory

12/07/2010 13:32:13|worker|host123|W|job 79.1 failed on host host123.epistemy.com general changing into working directory because: 12/07/2010 13:32:12 [1024:11078]: error: can't chdir to SGE-test: No such file or directory

On the positive side, "/SGE-test" works, though that isn't where I want to run the job.

Scott Wilson
Epistemy
Well, then. Last suggestion: don't use anything. The default assumption is that your starting point for all paths is the user's home directory. If you set the workingDirectory to ./SGE-test, it should resolve to ~/SGE-test.
Daniel
No bother. If only I got a penny every time *I* missed something. ;-)
12/07/2010 13:15:28|worker|host123|W|job 77.1 failed on host host123.epistemy.com general changing into working directory because: 12/07/2010 13:15:27 [1024:11046]: error: can't chdir to /SGE-test: No such file or directory
Is expansion of '~' not a shell feature, too?
Scott
Scott Wilson
Epistemy
Sorry, missed that. I'm answering emails from a hotel room while listening to a conference call. ;)
In that case, yes that should concern you. For now, try just using ~ instead of the placeholder. I'll try to reproduce it and file an issue if needed.
Daniel
I am using the JobTemplate.HOME_DIRECTORY placeholder. See code at the end of initial post. The $HOME is coming from somewhere in Grid Engine.
Scott
Scott Wilson
Epistemy
http://gridengine.sunsource.net/nonav/source/browse/~checkout~/gridengine/doc/javadocs/org/ggf/drmaa/JobTemplate.html#HOME_DIRECTORY
like s.setWorkingDirectory(JobTemplate.HOME_DIRECTORY + "/SGE-test")
Daniel
Post by epistemyscott
Dear all,
I'm trying to use the Java DRMAA bindings to run jobs, and I'm running into problems when trying to redirect output.
12/07/2010 11:19:28|worker|host123|W|job 71.1 failed on host host123.epistemy.com general opening input/output file because: 12/07/2010 11:19:27 [1024:14650]: error: can't open output file "$HOME/SGE-test/./SGE-test.log": No such file or directory
Touching the output files beforehand makes no difference.
12/07/2010 11:32:58|worker|host123|W|job 72.1 failed on host host123.epistemy.com general opening input/output file because: 12/07/2010 11:32:57 [1024:14705]: error: can't open output file "$HOME/SGE-test": No such file or directory
If I comment out the setWorkingDirectory line as well, it runs fine, but, of course, it puts all its output into my home directory with the wrong file names.
Should I be concerned at the apparent failure to expand $HOME, or is that just for logging purposes? Why does the second version seem to be trying to write its output directly onto the working directory? Does anyone have any suggestions on how to debug this? I'm running this using Java 1.6.0_17 on CentOS 5.5 and SGE 6.2u5.
Your help would be most appreciated.
Regards,
Scott
public static void submitJob(){
File runFolder = new File(System.getProperty("user.home"), "SGE-test");
runFolder.mkdir();
System.out.println("Folder created");
SessionFactory factory = SessionFactory.getFactory();
Session session = factory.getSession();
try{
session.init("");
JobTemplate job = session.createJobTemplate();
job.setRemoteCommand("pwd");
job.setJobName("SGE-test");
job.setBlockEmail(true);
job.setWorkingDirectory(JobTemplate.HOME_DIRECTORY+"SGE-test");
System.out.println("Working directory is: " + job.getWorkingDirectory());
job.setOutputPath(":"+JobTemplate.WORKING_DIRECTORY+"SGE-test.log");
job.setErrorPath(":"+JobTemplate.WORKING_DIRECTORY+"SGE-test.err.log");
String identifier = session.runJob(job);
System.out.println("Job submitted with identifier '" + identifier + "'.");
session.deleteJobTemplate(job);
}
catch(DrmaaException e){
System.out.println(e);
e.printStackTrace();
throw e;
}
finally{
try {
session.exit();
}
catch(DrmaaException e) {
System.out.println("Closing DRMAA session failed. " + e);
e.printStackTrace();
}
}
}
Scott Wilson
Developer
Epistemy
Energy Academy
Riccarton
EH14 4AS
tel: +44 131 564 0232
http://www.Epistemy.com/
Epistemy Limited is a company registered in Scotland, number SC365481.
Registered office: Epistemy Limited c/o Technology and Research Services, Heriot-Watt University, Riccarton, Edinburgh, EH14 4AS
------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302730
------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302777

To unsubscribe from this discussion, e-mail: [users-***@gridengine.sunsource.net].
epistemyscott
2010-12-07 13:21:00 UTC
Permalink
Raw Message
No bother. If only I got a penny every time *I* missed something. ;-)

Setting the working directory to "~/SGEtest" gets me:

12/07/2010 13:15:28|worker|host123|W|job 77.1 failed on host host123.epistemy.com general changing into working directory because: 12/07/2010 13:15:27 [1024:11046]: error: can't chdir to /SGE-test: No such file or directory

Is expansion of '~' not a shell feature, too?

Scott

Scott Wilson
Epistemy
Sorry, missed that. I'm answering emails from a hotel room while listening to a conference call. ;)
In that case, yes that should concern you. For now, try just using ~ instead of the placeholder. I'll try to reproduce it and file an issue if needed.
Daniel
I am using the JobTemplate.HOME_DIRECTORY placeholder. See code at the end of initial post. The $HOME is coming from somewhere in Grid Engine.
Scott
Scott Wilson
Epistemy
http://gridengine.sunsource.net/nonav/source/browse/~checkout~/gridengine/doc/javadocs/org/ggf/drmaa/JobTemplate.html#HOME_DIRECTORY
like s.setWorkingDirectory(JobTemplate.HOME_DIRECTORY + "/SGE-test")
Daniel
Post by epistemyscott
Dear all,
I'm trying to use the Java DRMAA bindings to run jobs, and I'm running into problems when trying to redirect output.
12/07/2010 11:19:28|worker|host123|W|job 71.1 failed on host host123.epistemy.com general opening input/output file because: 12/07/2010 11:19:27 [1024:14650]: error: can't open output file "$HOME/SGE-test/./SGE-test.log": No such file or directory
Touching the output files beforehand makes no difference.
12/07/2010 11:32:58|worker|host123|W|job 72.1 failed on host host123.epistemy.com general opening input/output file because: 12/07/2010 11:32:57 [1024:14705]: error: can't open output file "$HOME/SGE-test": No such file or directory
If I comment out the setWorkingDirectory line as well, it runs fine, but, of course, it puts all its output into my home directory with the wrong file names.
Should I be concerned at the apparent failure to expand $HOME, or is that just for logging purposes? Why does the second version seem to be trying to write its output directly onto the working directory? Does anyone have any suggestions on how to debug this? I'm running this using Java 1.6.0_17 on CentOS 5.5 and SGE 6.2u5.
Your help would be most appreciated.
Regards,
Scott
public static void submitJob(){
File runFolder = new File(System.getProperty("user.home"), "SGE-test");
runFolder.mkdir();
System.out.println("Folder created");
SessionFactory factory = SessionFactory.getFactory();
Session session = factory.getSession();
try{
session.init("");
JobTemplate job = session.createJobTemplate();
job.setRemoteCommand("pwd");
job.setJobName("SGE-test");
job.setBlockEmail(true);
job.setWorkingDirectory(JobTemplate.HOME_DIRECTORY+"SGE-test");
System.out.println("Working directory is: " + job.getWorkingDirectory());
job.setOutputPath(":"+JobTemplate.WORKING_DIRECTORY+"SGE-test.log");
job.setErrorPath(":"+JobTemplate.WORKING_DIRECTORY+"SGE-test.err.log");
String identifier = session.runJob(job);
System.out.println("Job submitted with identifier '" + identifier + "'.");
session.deleteJobTemplate(job);
}
catch(DrmaaException e){
System.out.println(e);
e.printStackTrace();
throw e;
}
finally{
try {
session.exit();
}
catch(DrmaaException e) {
System.out.println("Closing DRMAA session failed. " + e);
e.printStackTrace();
}
}
}
Scott Wilson
Developer
Epistemy
Energy Academy
Riccarton
EH14 4AS
tel: +44 131 564 0232
http://www.Epistemy.com/
Epistemy Limited is a company registered in Scotland, number SC365481.
Registered office: Epistemy Limited c/o Technology and Research Services, Heriot-Watt University, Riccarton, Edinburgh, EH14 4AS
------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302730
------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302772

To unsubscribe from this discussion, e-mail: [users-***@gridengine.sunsource.net].
reuti
2010-12-07 13:47:59 UTC
Permalink
Raw Message
No bother. If only I got a penny every time *I* missed something. ;-)
12/07/2010 13:15:28|worker|host123|W|job 77.1 failed on host host123.epistemy.com general changing into working directory because: 12/07/2010 13:15:27 [1024:11046]: error: can't chdir to /SGE-test: No such file or directory
Is expansion of '~' not a shell feature, too?
Yes, I think so; but it's also allowed as pseudo variable (`man qsub`, section "-e").

-- Reuti
Scott
Scott Wilson
Epistemy
Sorry, missed that. I'm answering emails from a hotel room while listening to a conference call. ;)
In that case, yes that should concern you. For now, try just using ~ instead of the placeholder. I'll try to reproduce it and file an issue if needed.
Daniel
I am using the JobTemplate.HOME_DIRECTORY placeholder. See code at the end of initial post. The $HOME is coming from somewhere in Grid Engine.
Scott
Scott Wilson
Epistemy
http://gridengine.sunsource.net/nonav/source/browse/~checkout~/gridengine/doc/javadocs/org/ggf/drmaa/JobTemplate.html#HOME_DIRECTORY
like s.setWorkingDirectory(JobTemplate.HOME_DIRECTORY + "/SGE-test")
Daniel
Post by epistemyscott
Dear all,
I'm trying to use the Java DRMAA bindings to run jobs, and I'm running into problems when trying to redirect output.
12/07/2010 11:19:28|worker|host123|W|job 71.1 failed on host host123.epistemy.com general opening input/output file because: 12/07/2010 11:19:27 [1024:14650]: error: can't open output file "$HOME/SGE-test/./SGE-test.log": No such file or directory
Touching the output files beforehand makes no difference.
12/07/2010 11:32:58|worker|host123|W|job 72.1 failed on host host123.epistemy.com general opening input/output file because: 12/07/2010 11:32:57 [1024:14705]: error: can't open output file "$HOME/SGE-test": No such file or directory
If I comment out the setWorkingDirectory line as well, it runs fine, but, of course, it puts all its output into my home directory with the wrong file names.
Should I be concerned at the apparent failure to expand $HOME, or is that just for logging purposes? Why does the second version seem to be trying to write its output directly onto the working directory? Does anyone have any suggestions on how to debug this? I'm running this using Java 1.6.0_17 on CentOS 5.5 and SGE 6.2u5.
Your help would be most appreciated.
Regards,
Scott
public static void submitJob(){
File runFolder = new File(System.getProperty("user.home"), "SGE-test");
runFolder.mkdir();
System.out.println("Folder created");
SessionFactory factory = SessionFactory.getFactory();
Session session = factory.getSession();
try{
session.init("");
JobTemplate job = session.createJobTemplate();
job.setRemoteCommand("pwd");
job.setJobName("SGE-test");
job.setBlockEmail(true);
job.setWorkingDirectory(JobTemplate.HOME_DIRECTORY+"SGE-test");
System.out.println("Working directory is: " + job.getWorkingDirectory());
job.setOutputPath(":"+JobTemplate.WORKING_DIRECTORY+"SGE-test.log");
job.setErrorPath(":"+JobTemplate.WORKING_DIRECTORY+"SGE-test.err.log");
String identifier = session.runJob(job);
System.out.println("Job submitted with identifier '" + identifier + "'.");
session.deleteJobTemplate(job);
}
catch(DrmaaException e){
System.out.println(e);
e.printStackTrace();
throw e;
}
finally{
try {
session.exit();
}
catch(DrmaaException e) {
System.out.println("Closing DRMAA session failed. " + e);
e.printStackTrace();
}
}
}
Scott Wilson
Developer
Epistemy
Energy Academy
Riccarton
EH14 4AS
tel: +44 131 564 0232
http://www.Epistemy.com/
Epistemy Limited is a company registered in Scotland, number SC365481.
Registered office: Epistemy Limited c/o Technology and Research Services, Heriot-Watt University, Riccarton, Edinburgh, EH14 4AS
------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302730
------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302772
------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302780

To unsubscribe from this discussion, e-mail: [users-***@gridengine.sunsource.net].
epistemyscott
2010-12-07 13:01:02 UTC
Permalink
Raw Message
I am using the JobTemplate.HOME_DIRECTORY placeholder. See code at the end of initial post. The $HOME is coming from somewhere in Grid Engine.

Scott

Scott Wilson
Epistemy
http://gridengine.sunsource.net/nonav/source/browse/~checkout~/gridengine/doc/javadocs/org/ggf/drmaa/JobTemplate.html#HOME_DIRECTORY
like s.setWorkingDirectory(JobTemplate.HOME_DIRECTORY + "/SGE-test")
Daniel
Post by epistemyscott
Dear all,
I'm trying to use the Java DRMAA bindings to run jobs, and I'm running into problems when trying to redirect output.
12/07/2010 11:19:28|worker|host123|W|job 71.1 failed on host host123.epistemy.com general opening input/output file because: 12/07/2010 11:19:27 [1024:14650]: error: can't open output file "$HOME/SGE-test/./SGE-test.log": No such file or directory
Touching the output files beforehand makes no difference.
12/07/2010 11:32:58|worker|host123|W|job 72.1 failed on host host123.epistemy.com general opening input/output file because: 12/07/2010 11:32:57 [1024:14705]: error: can't open output file "$HOME/SGE-test": No such file or directory
If I comment out the setWorkingDirectory line as well, it runs fine, but, of course, it puts all its output into my home directory with the wrong file names.
Should I be concerned at the apparent failure to expand $HOME, or is that just for logging purposes? Why does the second version seem to be trying to write its output directly onto the working directory? Does anyone have any suggestions on how to debug this? I'm running this using Java 1.6.0_17 on CentOS 5.5 and SGE 6.2u5.
Your help would be most appreciated.
Regards,
Scott
public static void submitJob(){
File runFolder = new File(System.getProperty("user.home"), "SGE-test");
runFolder.mkdir();
System.out.println("Folder created");
SessionFactory factory = SessionFactory.getFactory();
Session session = factory.getSession();
try{
session.init("");
JobTemplate job = session.createJobTemplate();
job.setRemoteCommand("pwd");
job.setJobName("SGE-test");
job.setBlockEmail(true);
job.setWorkingDirectory(JobTemplate.HOME_DIRECTORY+"SGE-test");
System.out.println("Working directory is: " + job.getWorkingDirectory());
job.setOutputPath(":"+JobTemplate.WORKING_DIRECTORY+"SGE-test.log");
job.setErrorPath(":"+JobTemplate.WORKING_DIRECTORY+"SGE-test.err.log");
String identifier = session.runJob(job);
System.out.println("Job submitted with identifier '" + identifier + "'.");
session.deleteJobTemplate(job);
}
catch(DrmaaException e){
System.out.println(e);
e.printStackTrace();
throw e;
}
finally{
try {
session.exit();
}
catch(DrmaaException e) {
System.out.println("Closing DRMAA session failed. " + e);
e.printStackTrace();
}
}
}
Scott Wilson
Developer
Epistemy
Energy Academy
Riccarton
EH14 4AS
tel: +44 131 564 0232
http://www.Epistemy.com/
Epistemy Limited is a company registered in Scotland, number SC365481.
Registered office: Epistemy Limited c/o Technology and Research Services, Heriot-Watt University, Riccarton, Edinburgh, EH14 4AS
------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302730
------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302765

To unsubscribe from this discussion, e-mail: [users-***@gridengine.sunsource.net].
templedf
2010-12-07 12:54:36 UTC
Permalink
Raw Message
When you're using DRMAA, there's no shell to expand $HOME for you.
It's being taken literally, e.g. \$HOME. There are two ways to
reference the user's home directory from the workingDirectory,
inputPath, and outputPath properties. The first is just to use standard
UNIX conventions, e.g. ~. The second is to use the
JobTemplate.HOME_DIRECTORY placeholder:

http://gridengine.sunsource.net/nonav/source/browse/~checkout~/gridengine/doc/javadocs/org/ggf/drmaa/JobTemplate.html#HOME_DIRECTORY

like s.setWorkingDirectory(JobTemplate.HOME_DIRECTORY + "/SGE-test")

Daniel
Post by epistemyscott
Dear all,
I'm trying to use the Java DRMAA bindings to run jobs, and I'm running into problems when trying to redirect output.
12/07/2010 11:19:28|worker|host123|W|job 71.1 failed on host host123.epistemy.com general opening input/output file because: 12/07/2010 11:19:27 [1024:14650]: error: can't open output file "$HOME/SGE-test/./SGE-test.log": No such file or directory
Touching the output files beforehand makes no difference.
12/07/2010 11:32:58|worker|host123|W|job 72.1 failed on host host123.epistemy.com general opening input/output file because: 12/07/2010 11:32:57 [1024:14705]: error: can't open output file "$HOME/SGE-test": No such file or directory
If I comment out the setWorkingDirectory line as well, it runs fine, but, of course, it puts all its output into my home directory with the wrong file names.
Should I be concerned at the apparent failure to expand $HOME, or is that just for logging purposes? Why does the second version seem to be trying to write its output directly onto the working directory? Does anyone have any suggestions on how to debug this? I'm running this using Java 1.6.0_17 on CentOS 5.5 and SGE 6.2u5.
Your help would be most appreciated.
Regards,
Scott
public static void submitJob(){
File runFolder = new File(System.getProperty("user.home"), "SGE-test");
runFolder.mkdir();
System.out.println("Folder created");
SessionFactory factory = SessionFactory.getFactory();
Session session = factory.getSession();
try{
session.init("");
JobTemplate job = session.createJobTemplate();
job.setRemoteCommand("pwd");
job.setJobName("SGE-test");
job.setBlockEmail(true);
job.setWorkingDirectory(JobTemplate.HOME_DIRECTORY+"SGE-test");
System.out.println("Working directory is: " + job.getWorkingDirectory());
job.setOutputPath(":"+JobTemplate.WORKING_DIRECTORY+"SGE-test.log");
job.setErrorPath(":"+JobTemplate.WORKING_DIRECTORY+"SGE-test.err.log");
String identifier = session.runJob(job);
System.out.println("Job submitted with identifier '" + identifier + "'.");
session.deleteJobTemplate(job);
}
catch(DrmaaException e){
System.out.println(e);
e.printStackTrace();
throw e;
}
finally{
try {
session.exit();
}
catch(DrmaaException e) {
System.out.println("Closing DRMAA session failed. " + e);
e.printStackTrace();
}
}
}
Scott Wilson
Developer
Epistemy
Energy Academy
Riccarton
EH14 4AS
tel: +44 131 564 0232
http://www.Epistemy.com/
Epistemy Limited is a company registered in Scotland, number SC365481.
Registered office: Epistemy Limited c/o Technology and Research Services, Heriot-Watt University, Riccarton, Edinburgh, EH14 4AS
------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302730
------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=302763

To unsubscribe from this discussion, e-mail: [users-***@gridengine.sunsource.net].
Loading...