Clustering Infrastructure

[_stanley]

I have been reading up on PVM (Parallel Virtual Machine) and MPI (Message Passing Interface). Both are used in Beowulf Clusters to 'simplify' clustering. The thing is, they both look terribly difficult, and I wish to make a cluster. So my plan is to:

- Write a clustering infrastructure that is much simpler than the above.
- Install Debian on the spare computer.
- Install the clustering software on as many computers as I am allowed to borrow steal.
- Find a compute-intensive algorithm.
- Write my own implemenation using my clustering software.
- Run it on the cluster and be happy.
- Sell my software to big corporations that wish to do lots of computerised thinking and become rich!

Actually, scrub the last one. It's GPL'd. Anyway, what do you think?
The clustering software will just be a small library that manages hosts connecting and thinking. Each host will know about all the other hosts. The root host will know what each host is doing. I might have a bunch of functions like:
int announce_host() - Announces a host on the network
int find_best_host(int important) - Finds the host that could do a calculation in the best time based on the parameter (e.g. which of connection speed, RAM, clock speed, etc. are most important)
int make_host_think(int hostnum, JOB jobdata) - Sends hostnum jobdata.
JOB get_job() - Waits until the local host is sent a job, returns what was sent with jobdata.
int new_host() - Returns hostid if a new host connects. Returns 0 if nothing
JOB get_result(JOB jobdata) - Returns the new jobdata if it has been computed, or NULL if the compute host hasn't finished yet
int send_result(JOB jobdata) - Sends jobdata back to the root host

struct JOB {
int id;//Integer containing job id
char name[32];//Char array containing job name
char chardata[1024];//Char array containing any char data
int intdata[1024];//Int array containing any int data
float floatdata[1024];//Float array containing any float data
}

There will probably be more stuff needed as I go along.

Anyway, what does anyone think?
All I'm really doing is having a place to post my idea. I doubt anyone will bother to read it all.

EDIT:
Example,
root_host.c:

//Usual init stuff

#define CONNECT_SPEED 0
//So that the reader knows what it means when I pass 0 to find_best_host()

int hostid;
JOB tempry;
JOB example_job;

JOB->id = 1;
JOB->name = "Example";//I don't know if char arrays can be treated like this, we'll see...
//Assign some example data:
JOB->chardata[0] = 'a';
JOB->intdata[0] = 6;
JOB->floatdata[0] = 65.776;

while(!new_host());//Wait until at least one other host connects

make_host_think(find_best_host(CONNECT_SPEED), example_job);//Send the best host based on connection speed the example job we made earlier. We ignore the returned error code, we just assume it was sent properly

while(1) {
tempry = get_result(example_job);//Get the result of example_job
if(tempry) break;//If we got a result, break out of the while loop
}

//Now we just act on tempry.
//A system like this could be used with any number of jobs and hosts.


compute_host.c:

//Usual init stuff

JOB my_job;//Don't assign anything to it as we get it from get_job()

if(!announce_host()) exit(1);//No root host found

start:
my_job = get_job();//Wait until root hosts picks us and sends us a job

if(my_job->textdata[0] = 'a') {
my_job->textdata = "Job done!";
} else {
my_job->textdata = "Job done, but textdata[0] wasn't a";
}

send_result(my_job);//Return the result to the root host
goto start;


EDIT2:
I will probably also have more stuff to do with which hosts are allowed, and checking an ip specifically, in case they can't announce themselves (e.g. blocked by firewall, over internet).

[Maximilian]

Finish Stick Territory first, eh?

admin said:

- Install the clustering software on as many computers as I am allowed to borrow steal.

There won't be any many computers then. Stealing is illegal.

[_stanley]

No! Never!
I am going to do Red and Blue skins tomorrow, then add support for changing of the skin, then release it on Christmas Day.

EDIT:
I will be able to run it on:
- My main one
- My 'custom' motherboard one
- My laptop

hopefully:
- My Dad's computer
- My Mum's computer

EDIT2:
I will probably do more thinking about how data is contained in JOB.

[Maximilian]

Will you be able to run it over SSH ?

[_stanley]

What? The clustering? Yeah, why wouldn't you?

[Maximilian]

Mwahahahahahaha!

[_stanley]

Oh, I see! You are going to hacksaw haxor a load of insecure Windows machines, install an SSH server and SCI (Stanley Cluster Infrastructure) on them, and then use them to calculate the best defense strategy for OGame? Clever!

[Maximilian]

I just wanted to play a game of Klondike.

[_stanley]

What? What does SSH have to do with that?

EDIT:
I think I might revise my plans about it and start coding soon. Also, I will do more work on ST.
Next thing to do is a bug in the panel. All the names show up in the colour of the current player, not the colour of the team that player is on. Also, if the player is on a team other Red or Blue (impossible - hopefully! - at the moment), then a lot of strange stuff will happen with the colours... As in, the colour variable will stay uninitialised.

From player.cpp, draw_panel():

//Line 358
 if(teams[0] == BLUETEAM) colour = blue; else colour = red;
 textprintf_ex(bmp, font, 4, 432, colour, -1, "%04d %s", points[0], names[0]);
//If not blue, do it in red...



//Line 371
     switch(player->team) {
       case BLUETEAM:
         textprintf_ex(bmp, font, 156, 386, blue, -1, "Current player: %s", player->name);
         break;
       case REDTEAM:
         textprintf_ex(bmp, font, 156, 386, red, -1, "Current player: %s", player->name);
         break;
     }
//If not blue or red, just don't bother :-/


And there's plenty more where that came from. I'll spend most of my time on my computer in the Christmas Holidays finding little things like this.

EDIT2:
If you check the latest source you will see that the code is actually tabulated properly. It's just the code tags in this don't actually let you type code properly... Also, the line numbers are roughly correct (they may not be exact in the last uploaded source...).

[Maximilian]

SSH + VNC + IP address and other details = Klondike

[_stanley]

Install an X server on your computer and do:
ssh -C -X james@82.33.119.96
enter my password
type kolf
Play mini golf on my computer from your computer with X forwarding!

[Maximilian]

x.cygwin.com/

Hmm... will I be able to use this ?
(I really don't want to spend an hour installing the w/lan driver for Knoppix, or burning a MEPIS disk and hoping that it works)

[_stanley]

I meant something like Cygwin/X or Xming, yeah.
Also, I have started learning UNIX sockets (BDNE was not powerful enough) - this means that unless somebody can port it to Windows, SCI will be POSIX-only. Same for StickTerritory.

[Maximilian]

admin said:

this means that unless somebody can port it to Windows, SCI will be POSIX-only. Same for StickTerritory.

Time to learn WinSock all over again...

[_stanley]

Indeed.
Hopefully, someone on Allegro.cc will help me. I'm not spending too long on it, though. To be honest, I don't care about Windows.
Or, you could install the Cygwin dev. libraries, and compile it with POSIX.

EDIT:
I know what I'm going to write as well. I shall do either a lottery or football simulation to calculate the outcomes.
The lottery one would be easier to write, but harder to get right, because it is mostly random. The opposite for the football, as it involves skill. I could have tables of the results between each teams over the last few years, make it count the more recent ones as more important, and have it work out what the score is most likely to be.

[_stanley]

OK, I have written a just-about-useable Telnet client and joined a MUD called Asylum.
The only problems with my client are:
- Not capable of displaying colour (not important)
- It won't print out a line until an end line character is recieved (only in proper Telnet Servers, it works with my one). VERY important, as 'Username: ' doesn't have an end line at the end.
- Echoes text whether the server tells it to or not (e.g. in password fields). Most servers let the client think about whether to echo or not, so this isn't too bad.

I was quite pleased, anyway I have just started the SCI code.

[Maximilian]

You wrote it using proper sockets ?

[_stanley]

Yep!
Here's the code:

/*A simple telnet-like client

By James Stanley*/

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <netdb.h>
#include <fcntl.h>
#include <signal.h>
#include <sys/wait.h>

void sigchld_handler(int s) {
 while(waitpid(-1, NULL, WNOHANG) > 0);
}

int main(int argc, char *argv[]) {
 int socketfd;
 struct hostent *server;
 struct sigaction zombiekill;
 struct sockaddr_in servaddr;
 char senddata[128];
 char recvdata[128];
 int n, port = 0;
 char *quit = "0";
 
 if(argc != 3) {
   printf("./client host port\n");
   exit(1);
 }
 
 server = gethostbyname(argv[1]);
 if(server == NULL) {
   printf("Error finding host: %s\n", argv[1]);
   exit(1);
 }
 
 socketfd = socket(PF_INET, SOCK_STREAM, 0);
 if(socketfd == -1) {
   printf("Error creating socket\n");
   exit(1);
 }
 
 port = atoi(argv[2]);
 
 servaddr.sin_family = AF_INET;
 servaddr.sin_port = htons(port); //NBO SHORT
 servaddr.sin_addr = *((struct in_addr *)server->h_addr);
 memset(&(servaddr.sin_zero), '\0', 8);
 
 n = connect(socketfd, (struct sockaddr *)&servaddr, sizeof(struct sockaddr));
 if(n == -1) {
   printf("Error connecting to %s on port %d\n", inet_ntoa(servaddr.sin_addr), port);
   exit(1);
 }
 
 zombiekill.sa_handler = sigchld_handler;//Kill zombie processes
 sigemptyset(&zombiekill.sa_mask);
 zombiekill.sa_flags = SA_RESTART;
 n = sigaction(SIGCHLD, &zombiekill, NULL);
 if(n == -1) {
   printf("Error setting up zombie killer");
   exit(1);
 }
 
 n = fork();//A nasty hack so that we can use (blocking) fgets and recv - They only block one thread each :)
 
 if(!n) {//Child process
   while(1) {
     n = recv(socketfd, recvdata, 127, 0);
     if(n != 0) {
       recvdata[n] = '\0';
       printf("%s", recvdata);
       memset(&recvdata, '\0', 128);
     } else {
       //Error
       printf("\n\n\nConnection closed by remote host. Press ENTER a few times.\n");
       quit = "1";
       exit(1);
     }
   }
 } else {//Parent process
   while(1) {
     if(strcmp(quit, "1") == 0) exit(1);
     fgets(senddata, 128, stdin);
     n = send(socketfd, senddata, strlen(senddata), 0);
     if(n == -1) {
       printf("Error sending %s\n", senddata);
       close(socketfd);
       exit(1);
     }
   }
 }
 
 close(socketfd);
 return(0);
}

I used this tutorial to learn sockets:
beej.us/guide/bgnet/output/html/index.html

EDIT:
The real one is properly indented, it's just that ProBoards don't actually think that people might want their code properly tabulated when they put it in code tags...

[Maximilian]

That site.. I've used it before on my quest to find out how to program WinSock.
A link from this guys website lead me to a great place to learn about WinSock... unfortunately I'm rather annoyed of the errors when compiling... and WinSock is soooo much easier in VB (I think the OCX file is all the C code ready and waiting)

[_stanley]

It's not all the C code. It's only a small section of what can be achieved. It's about what you need to write a client like what I posted above. A cluster infrastructure written in VB would not be good.

[blaize]

I don't think Max was around to experience Stanley's last attempt at clustering. It ended as no-one would donate their computer.

Good luck this time

[Maximilian]

It was only last year (I think)..

clusterKnoppix I think it was? Or was it the other one whose name I cannot remember?

[_stanley]

I don't need anyones computer. I already have some. It ended because I wasn't prepared to install a 2.4 kernel just to get OpenMosix. Now I am writing my own software.

[blaize]

Oh. Cool

[Maximilian]

admin said:

I don't need anyones computer. I already have some. It ended because I wasn't prepared to install a 2.4 kernel just to get OpenMosix. Now I am writing my own software.

That's the one.