Post by _stanley on Dec 23, 2006 19:30:01 GMT
I have been reading up on PVM (Parallel Virtual Machine) and MPI (Message Passing Interface). Both are used in Beowulf Clusters to 'simplify' clustering. The thing is, they both look terribly difficult, and I wish to make a cluster. So my plan is to:
- Write a clustering infrastructure that is much simpler than the above.
- Install Debian on the spare computer.
- Install the clustering software on as many computers as I am allowed toborrow steal.
- Find a compute-intensive algorithm.
- Write my own implemenation using my clustering software.
- Run it on the cluster and be happy.
- Sell my software to big corporations that wish to do lots of computerised thinking and become rich!
Actually, scrub the last one. It's GPL'd. Anyway, what do you think?
The clustering software will just be a small library that manages hosts connecting and thinking. Each host will know about all the other hosts. The root host will know what each host is doing. I might have a bunch of functions like:
int announce_host() - Announces a host on the network
int find_best_host(int important) - Finds the host that could do a calculation in the best time based on the parameter (e.g. which of connection speed, RAM, clock speed, etc. are most important)
int make_host_think(int hostnum, JOB jobdata) - Sends hostnum jobdata.
JOB get_job() - Waits until the local host is sent a job, returns what was sent with jobdata.
int new_host() - Returns hostid if a new host connects. Returns 0 if nothing
JOB get_result(JOB jobdata) - Returns the new jobdata if it has been computed, or NULL if the compute host hasn't finished yet
int send_result(JOB jobdata) - Sends jobdata back to the root host
struct JOB {
int id;//Integer containing job id
char name[32];//Char array containing job name
char chardata[1024];//Char array containing any char data
int intdata[1024];//Int array containing any int data
float floatdata[1024];//Float array containing any float data
}
There will probably be more stuff needed as I go along.
Anyway, what does anyone think?
All I'm really doing is having a place to post my idea. I doubt anyone will bother to read it all.
EDIT:
Example,
root_host.c:
compute_host.c:
EDIT2:
I will probably also have more stuff to do with which hosts are allowed, and checking an ip specifically, in case they can't announce themselves (e.g. blocked by firewall, over internet).
- Write a clustering infrastructure that is much simpler than the above.
- Install Debian on the spare computer.
- Install the clustering software on as many computers as I am allowed to
- Find a compute-intensive algorithm.
- Write my own implemenation using my clustering software.
- Run it on the cluster and be happy.
- Sell my software to big corporations that wish to do lots of computerised thinking and become rich!
Actually, scrub the last one. It's GPL'd. Anyway, what do you think?
The clustering software will just be a small library that manages hosts connecting and thinking. Each host will know about all the other hosts. The root host will know what each host is doing. I might have a bunch of functions like:
int announce_host() - Announces a host on the network
int find_best_host(int important) - Finds the host that could do a calculation in the best time based on the parameter (e.g. which of connection speed, RAM, clock speed, etc. are most important)
int make_host_think(int hostnum, JOB jobdata) - Sends hostnum jobdata.
JOB get_job() - Waits until the local host is sent a job, returns what was sent with jobdata.
int new_host() - Returns hostid if a new host connects. Returns 0 if nothing
JOB get_result(JOB jobdata) - Returns the new jobdata if it has been computed, or NULL if the compute host hasn't finished yet
int send_result(JOB jobdata) - Sends jobdata back to the root host
struct JOB {
int id;//Integer containing job id
char name[32];//Char array containing job name
char chardata[1024];//Char array containing any char data
int intdata[1024];//Int array containing any int data
float floatdata[1024];//Float array containing any float data
}
There will probably be more stuff needed as I go along.
Anyway, what does anyone think?
All I'm really doing is having a place to post my idea. I doubt anyone will bother to read it all.
EDIT:
Example,
root_host.c:
//Usual init stuff
#define CONNECT_SPEED 0
//So that the reader knows what it means when I pass 0 to find_best_host()
int hostid;
JOB tempry;
JOB example_job;
JOB->id = 1;
JOB->name = "Example";//I don't know if char arrays can be treated like this, we'll see...
//Assign some example data:
JOB->chardata[0] = 'a';
JOB->intdata[0] = 6;
JOB->floatdata[0] = 65.776;
while(!new_host());//Wait until at least one other host connects
make_host_think(find_best_host(CONNECT_SPEED), example_job);//Send the best host based on connection speed the example job we made earlier. We ignore the returned error code, we just assume it was sent properly
while(1) {
tempry = get_result(example_job);//Get the result of example_job
if(tempry) break;//If we got a result, break out of the while loop
}
//Now we just act on tempry.
//A system like this could be used with any number of jobs and hosts.
compute_host.c:
//Usual init stuff
JOB my_job;//Don't assign anything to it as we get it from get_job()
if(!announce_host()) exit(1);//No root host found
start:
my_job = get_job();//Wait until root hosts picks us and sends us a job
if(my_job->textdata[0] = 'a') {
my_job->textdata = "Job done!";
} else {
my_job->textdata = "Job done, but textdata[0] wasn't a";
}
send_result(my_job);//Return the result to the root host
goto start;
EDIT2:
I will probably also have more stuff to do with which hosts are allowed, and checking an ip specifically, in case they can't announce themselves (e.g. blocked by firewall, over internet).