[hobbit] multiple hosts
I know for us here (running trucluster 5) we ran into the same problem, here is my modified runclient.sh script (the changes are basically I added the hostname part to the lock files:
#!/bin/sh #----------------------------------------------------------------------------#
Hobbit client bootup script.
This invokes hobbitlaunch, which in turn runs the Hobbit client and any
extensions configured.
Copyright (C) 2005 Henrik Storner <henrik at hswn.dk>
This program is released under the GNU General Public License (GPL),
version 2. See the file "COPYING" for details.
#----------------------------------------------------------------------------#
$Id: runclient.sh,v 1.6 2005/09/28 21:21:17 henrik Exp $
Default settings for this client
MACHINEDOTS="uname -n" # This systems hostname
BBOSTYPE="uname -s | tr '[A-Z]' '[a-z]'" # This systems operating system in lowercase
BBOSSCRIPT="hobbitclient-$BBOSTYPE.sh"
Commandline mods for the defaults
while test "$1" != ""
do
case "$1" in
--hostname=*)
MACHINEDOTS="echo $1 | sed -e's/--hostname=//'"
;;
--os=*)
BBOSTYPE="echo $1 | sed -e's/--os=//' | tr '[A-Z]' '[a-z]'"
;;
--help)
echo "Usage: $0 [--hostname=CLIENTNAME] [--os=rhel3|linux22] start|stop"
exit 0
;;
start)
CMD=$1
;;
stop)
CMD=$1
;;
restart)
CMD=$1
;;
esac
shiftdone
OLDDIR="pwd"
cd "dirname $0"
HOBBITCLIENTHOME="pwd"
cd "$OLDDIR"
MACHINE="echo $MACHINEDOTS | sed -e's/\./,/g'"
export MACHINE MACHINEDOTS BBOSTYPE BBOSSCRIPT HOBBITCLIENTHOME
case "$CMD" in "start") if test ! -w $HOBBITCLIENTHOME/logs; then echo "Cannot write to the $HOBBITCLIENTHOME/logs directory" exit 1 fi if test ! -w $HOBBITCLIENTHOME/tmp; then echo "Cannot write to the $HOBBITCLIENTHOME/tmp directory" exit 1 fi
if test -f $HOBBITCLIENTHOME/logs/clientlaunch.`hostname`.pid; then
echo "Hobbit client already running, re-starting it"
$0 stop
rm -f $HOBBITCLIENTHOME/logs/clientlaunch.`hostname`.pid
fi
$HOBBITCLIENTHOME/bin/hobbitlaunch --config=$HOBBITCLIENTHOME/etc/clientlaunch.cfg --log=$HOBBITCLIENTHOME/logs/clientlaunch.log --pidfile=$HOBBITCLIENTHOME/logs/clientlaunch.`hostname`.pid
if test $? -eq 0; then
echo "Hobbit client for $BBOSTYPE started on $MACHINEDOTS"
else
echo "Hobbit client startup failed"
fi
;;"stop")
if test -f $HOBBITCLIENTHOME/logs/clientlaunch.hostname.pid; then
kill cat $HOBBITCLIENTHOME/logs/clientlaunch.$MACHINEDOTS.pid
echo "Hobbit client stopped"
else
echo "Hobbit client not running"
fi
;;
"restart")
if test -f $HOBBITCLIENTHOME/logs/clientlaunch.hostname.pid; then
$0 stop
else
echo "Hobbit client not running, continuing to start it"
fi
$0 start
;;esac
exit 0
-----Original Message----- From: David Gore [mailto:David.Gore at mci.com] Sent: Wednesday, January 11, 2006 9:52 AM To: hobbit at hswn.dk Subject: Re: [hobbit] multiple hosts
Rolf,
I asked a similar question many months ago for our Digital UNIX (4.x) boxes running in multiple clusters. I did not get any answer, so I had to create a hack that does run multiple clients on each host. One client is for the physical host and does, cpu, disk, etc, and one client for the logical/virtual host that does the tasks specific to the logical/virtual host. The runclient.sh script runs every ten minutes via crontab, to make sure both clients are running and starts either client if they are not. I also have custom versions of some of the default Hobbit client scripts to support multiple clients on a single host. So in short as far as I know, you are going to have to hack/script your own solution. I don't mind sending you what my hobbit scripts have turned into, but it may not be what you are looking for.
David Gore
*e-mail via SUSE Linux 9.3 and other open source tools.
Rolf Schrittenlocher wrote:
Hi,
as there was no answer to my first mail, I try again with other words:
Our machines may have one or more aliases (virtual hosts) which all have DNS-entries. Normally we refer to the machines not using the hostname but by the alias, each of them combined to a bundle of functions and tasks. So a machine "frodo.xxx.de" might host "webserver.xxx.de", "dnsserver.xxx.de" and "tolkserver.xxx.de". These aliases and of course the tasks related to them are transferred to another machine in case of hardware failure, maintenance, etc. In bb-hosts we have an entry for each machine (to get data from this machine in case there is no alias set on it) and of course for all relevant aliases. What we need now is that the general client data (cpu, disk, ssh, ...) for each machine appears both with the hostname and all aliases set on it. I added a script to runclient.sh which creates ~/client/etc/clientlaunch.cfg dynamically according to the aliases set on the machine before starting the client. This assures that the correct data is polled on the client. I may modify $MACHINE or Í may use the CLIENT:hostname tag in bb-hosts to make the client send messages with a different name. But that would require several instances of the client running, somethin I dislike. I hope there is a way to send the client data once and to make the server look at some configuration file to see where he should present these data. Maybe that possibility exists but I haven't found it yet. So I hope that someone could indicate me where to search for the places which I have to "hack" hoping that it is possible to do that with shell or perl as my C-knowledge isn't so good. But even if C is necessary: Any hints are appreciated.
Thanks Rolf
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
So I'm staring at
- Program crashed Fatal signal caught!
from the purple hobbitd_client status for the server, which is very odd, considering the server (box itself, not just Hobbit binary...) has been completely restarted about an hour ago.
There were some horrible, horrible disk problems (many backups and a fsck.reiserfs later... don't ask) due to someone screwing building power whilst i was away and zapping this box with something like 5 restarts in half an hour. ugh.
However, having fixed everything else, and up and running from a clean boot, and ONLY running hobbit.sh from the /hobbit/server dir... i'm confused. Hobbitd col shows green. There's only one hobbitd_client running on the box, and it's the one that appeared when the server itself was started.
Why does this happen, and how can i stop it?
thanks...
The hobbitd_client only issues a status message when it dies. After a restart it will go purple in a half-hour. You just need to do a "drop" of the hobbitd service.
Thanks, Larry Barber
On 1/11/06, Rob Munsch <rmunsch at solutionsforprogress.com> wrote:
So I'm staring at
- Program crashed Fatal signal caught!
from the purple hobbitd_client status for the server, which is very odd, considering the server (box itself, not just Hobbit binary...) has been completely restarted about an hour ago.
There were some horrible, horrible disk problems (many backups and a fsck.reiserfs later... don't ask) due to someone screwing building power whilst i was away and zapping this box with something like 5 restarts in half an hour. ugh.
However, having fixed everything else, and up and running from a clean boot, and ONLY running hobbit.sh from the /hobbit/server dir... i'm confused. Hobbitd col shows green. There's only one hobbitd_client running on the box, and it's the one that appeared when the server itself was started.
Why does this happen, and how can i stop it?
thanks...
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
ooookay (and i assume you mean the hobbitd_client service), that does indeed make it Go Away - but, if it ever crashes again for any reason, it will reappear red then go perma-purple again, requiring another manual Drop, yes?
Surely there's a better way to handle this..?
Larry Barber wrote:
The hobbitd_client only issues a status message when it dies. After a restart it will go purple in a half-hour. You just need to do a "drop" of the hobbitd service.
Thanks, Larry Barber
On 1/11/06, *Rob Munsch* <rmunsch at solutionsforprogress.com <mailto:rmunsch at solutionsforprogress.com>> wrote:
So I'm staring at - Program crashed Fatal signal caught! from the purple hobbitd_client status for the server, which is very odd, considering the server (box itself, not just Hobbit binary...) has been completely restarted about an hour ago. There were some horrible, horrible disk problems (many backups and a fsck.reiserfs later... don't ask) due to someone screwing building power whilst i was away and zapping this box with something like 5 restarts in half an hour. ugh. However, having fixed everything else, and up and running from a clean boot, and ONLY running hobbit.sh from the /hobbit/server dir... i'm confused. Hobbitd col shows green. There's only one hobbitd_client running on the box, and it's the one that appeared when the server itself was started. Why does this happen, and how can i stop it? thanks... To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk <mailto:hobbit-unsubscribe at hswn.dk>
On Wed, Jan 11, 2006 at 02:52:58PM -0500, Rob Munsch wrote:
ooookay (and i assume you mean the hobbitd_client service), that does indeed make it Go Away - but, if it ever crashes again for any reason, it will reappear red then go perma-purple again, requiring another manual Drop, yes?
Surely there's a better way to handle this..?
Yes there is - fix the bug that makes hobbitd_client crash, and you won't see any hobbitd_client statuses. Ever.
For starters, check that you are running the latest version (4.1.2p1 - the older versions do have bugs in the hobbitd_client code). And check the "Reporting Bugs" part of http://www.hswn.dk/hobbit/help/known-issues.html
Regards, Henrik
gdb> bt #0 0xb7e5b941 in kill () from /lib/libc.so.6 #1 0xb7e5b6e5 in raise () from /lib/libc.so.6 #2 0xb7e5ca66 in abort () from /lib/libc.so.6 #3 0x08056651 in sigsegv_handler (signum=0x0) at sig.c:57 #4 <signal handler called> #5 0xb7e9930c in mempcpy () from /lib/libc.so.6 #6 0xb7e905a1 in _IO_default_xsputn () from /lib/libc.so.6 #7 0xb7e6ef60 in vfprintf () from /lib/libc.so.6 #8 0xb7e876ec in vsprintf () from /lib/libc.so.6 #9 0xb7e7561d in sprintf () from /lib/libc.so.6 #10 0x08049ed0 in unix_cpu_report (hostname=0x20202020 <Address 0x20202020 out of bounds>, hinfo=0x20c7, fromline=0x303a3030 <Address 0x303a3030 out of bounds>, timestr=0x34313a31 <Address 0x34313a31 out of bounds>, uptimestr=0x736f7020 <Address 0x736f7020 out of bounds>, whostr=0x65726774 <Address 0x65726774 out of bounds>, psstr=0x73203a73 <Address 0x73203a73 out of bounds>, topstr=0x6e616863 <Address 0x6e616863 out of bounds>) at hobbitd_client.c:242 #11 0x3f203430 in ?? () #12 0x20202020 in ?? () #13 0x20202020 in ?? ()
... at this point, please not that lines #14 - #2387 (!!) all look the same as the previous three - the address varies, but it's 2300+ lines of "<address> in ??." If that whole list is somehow useful, let me know.. otherwise i'll just cut to the end:
#2384 0x20203937 in ?? () #2385 0x614a2030 in ?? () #2386 0x2034306e in ?? () #2387 0x2020203f in ?? () Cannot access memory at address 0xbfea0000 gdb>
Henrik Stoerner wrote:
On Wed, Jan 11, 2006 at 02:52:58PM -0500, Rob Munsch wrote:
ooookay (and i assume you mean the hobbitd_client service), that does indeed make it Go Away - but, if it ever crashes again for any reason, it will reappear red then go perma-purple again, requiring another manual Drop, yes?
Surely there's a better way to handle this..?
Yes there is - fix the bug that makes hobbitd_client crash, and you won't see any hobbitd_client statuses. Ever.
For starters, check that you are running the latest version (4.1.2p1 - the older versions do have bugs in the hobbitd_client code). And check the "Reporting Bugs" part of http://www.hswn.dk/hobbit/help/known-issues.html
Regards, Henrik
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
I've been away for a week and no longer sure if this was updated. grr.
i'll have to check if this is -.2 or -.2p1 now. sigh...
Rob Munsch wrote:
gdb> bt #0 0xb7e5b941 in kill () from /lib/libc.so.6 #1 0xb7e5b6e5 in raise () from /lib/libc.so.6 #2 0xb7e5ca66 in abort () from /lib/libc.so.6 #3 0x08056651 in sigsegv_handler (signum=0x0) at sig.c:57 #4 <signal handler called> #5 0xb7e9930c in mempcpy () from /lib/libc.so.6 #6 0xb7e905a1 in _IO_default_xsputn () from /lib/libc.so.6 #7 0xb7e6ef60 in vfprintf () from /lib/libc.so.6 #8 0xb7e876ec in vsprintf () from /lib/libc.so.6 #9 0xb7e7561d in sprintf () from /lib/libc.so.6 #10 0x08049ed0 in unix_cpu_report (hostname=0x20202020 <Address 0x20202020 out of bounds>, hinfo=0x20c7, fromline=0x303a3030 <Address 0x303a3030 out of bounds>, timestr=0x34313a31 <Address 0x34313a31 out of bounds>, uptimestr=0x736f7020 <Address 0x736f7020 out of bounds>, whostr=0x65726774 <Address 0x65726774 out of bounds>, psstr=0x73203a73 <Address 0x73203a73 out of bounds>, topstr=0x6e616863 <Address 0x6e616863 out of bounds>) at hobbitd_client.c:242 #11 0x3f203430 in ?? () #12 0x20202020 in ?? () #13 0x20202020 in ?? ()
... at this point, please not that lines #14 - #2387 (!!) all look the same as the previous three - the address varies, but it's 2300+ lines of "<address> in ??." If that whole list is somehow useful, let me know.. otherwise i'll just cut to the end:
#2384 0x20203937 in ?? () #2385 0x614a2030 in ?? () #2386 0x2034306e in ?? () #2387 0x2020203f in ?? () Cannot access memory at address 0xbfea0000 gdb>
Henrik Stoerner wrote:
On Wed, Jan 11, 2006 at 02:52:58PM -0500, Rob Munsch wrote:
ooookay (and i assume you mean the hobbitd_client service), that does indeed make it Go Away - but, if it ever crashes again for any reason, it will reappear red then go perma-purple again, requiring another manual Drop, yes?
Surely there's a better way to handle this..?
Yes there is - fix the bug that makes hobbitd_client crash, and you won't see any hobbitd_client statuses. Ever.
For starters, check that you are running the latest version (4.1.2p1 - the older versions do have bugs in the hobbitd_client code). And check the "Reporting Bugs" part of http://www.hswn.dk/hobbit/help/known-issues.html
Regards, Henrik
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
What is the sound of one forehead slapping?
So yeah. Apparently this is an install of .2 that was done shortly before .2p1 was released, and i missed noticing the transition. So since my mind has apparently turned to mush in a mere week of offtime, before i b0rk this up further, are there any caveats or tricks to upgrading here, or do i just need to replace wholesale all the various components entirely..?
thanks,
rob
Rob Munsch wrote:
I've been away for a week and no longer sure if this was updated. grr. i'll have to check if this is -.2 or -.2p1 now. sigh...
...
For starters, check that you are running the latest version (4.1.2p1 -
On Wed, Jan 11, 2006 at 04:54:33PM -0500, Rob Munsch wrote:
What is the sound of one forehead slapping?
So yeah. Apparently this is an install of .2 that was done shortly before .2p1 was released, and i missed noticing the transition. So since my mind has apparently turned to mush in a mere week of offtime, before i b0rk this up further, are there any caveats or tricks to upgrading here, or do i just need to replace wholesale all the various components entirely..?
Doing a "make; make install" should be safe.
Regards, Henrik
really??? awesome! thanks!
Henrik Stoerner wrote:
On Wed, Jan 11, 2006 at 04:54:33PM -0500, Rob Munsch wrote:
What is the sound of one forehead slapping?
So yeah. Apparently this is an install of .2 that was done shortly before .2p1 was released, and i missed noticing the transition. So since my mind has apparently turned to mush in a mere week of offtime, before i b0rk this up further, are there any caveats or tricks to upgrading here, or do i just need to replace wholesale all the various components entirely..?
Doing a "make; make install" should be safe.
Regards, Henrik
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
Hi, I wonder if this is an issue with the server or with the client.
Im running the latest version of the server (4.1.2p1) and I still couldnt get rid off the alert.
Many thanks,
Fabio
-----Original Message----- From: Henrik Stoerner [mailto:henrik at hswn.dk] Sent: 11 January 2006 20:47 To: hobbit at hswn.dk Subject: Re: [hobbit] eternally purple hobbitd_client on the server box
On Wed, Jan 11, 2006 at 02:52:58PM -0500, Rob Munsch wrote:
ooookay (and i assume you mean the hobbitd_client service), that does indeed make it Go Away - but, if it ever crashes again for any reason, it will reappear red then go perma-purple again, requiring another manual Drop, yes?
Surely there's a better way to handle this..?
Yes there is - fix the bug that makes hobbitd_client crash, and you won't see any hobbitd_client statuses. Ever.
For starters, check that you are running the latest version (4.1.2p1 - the older versions do have bugs in the hobbitd_client code). And check the "Reporting Bugs" part of http://www.hswn.dk/hobbit/help/known-issues.html
Regards, Henrik
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
I do not think I address the original question, but we also run TruCluster5 and just use CDSL links for hobbitclient tmp and log directories. No script modifications were necessary. This is how hobbit client directory looks: drwxr-xr-x 2 hobbit hobbit 8192 Oct 17 10:01 bin drwxr-xr-x 2 hobbit hobbit 8192 Oct 17 10:01 etc drwxr-xr-x 2 hobbit hobbit 8192 Oct 17 10:01 ext lrwxrwxrwx 1 root hobbit 56 Oct 24 09:02 logs -> ../../../cluster/members/{memb}/local/hobbit/client/logs -r-xr-xr-x 1 hobbit users 3001 Sep 28 17:21 runclient.sh lrwxrwxrwx 1 root hobbit 55 Oct 24 09:02 tmp -> ../../../cluster/members/{memb}/local/hobbit/client/tmp
We only monitor cluster members at this point, not the virtual host (cluster aliases)
I do have one problem with Tru64. The disk utilization is being sometimes reported on device/domain basis, sometimes on the mount point basis HP-UX and Linux always reported as mountpoint. If anyone knows how address this, I will greatly appreciate it
Thanks,
Alex.
Scheblein, Adam wrote:
I know for us here (running trucluster 5) we ran into the same problem, here is my modified runclient.sh script (the changes are basically I added the
hostnamepart to the lock files:#!/bin/sh #----------------------------------------------------------------------------#
Hobbit client bootup script.
This invokes hobbitlaunch, which in turn runs the Hobbit client and any
extensions configured.
Copyright (C) 2005 Henrik Storner <henrik at hswn.dk>
This program is released under the GNU General Public License (GPL),
version 2. See the file "COPYING" for details.
#----------------------------------------------------------------------------#
$Id: runclient.sh,v 1.6 2005/09/28 21:21:17 henrik Exp $
Default settings for this client
MACHINEDOTS="
uname -n" # This systems hostname BBOSTYPE="uname -s | tr '[A-Z]' '[a-z]'" # This systems operating system in lowercase BBOSSCRIPT="hobbitclient-$BBOSTYPE.sh"Commandline mods for the defaults
while test "$1" != "" do case "$1" in --hostname=*) MACHINEDOTS="
echo $1 | sed -e's/--hostname=//'" ;; --os=*) BBOSTYPE="echo $1 | sed -e's/--os=//' | tr '[A-Z]' '[a-z]'" ;; --help) echo "Usage: $0 [--hostname=CLIENTNAME] [--os=rhel3|linux22] start|stop" exit 0 ;; start) CMD=$1 ;; stop) CMD=$1 ;; restart) CMD=$1 ;; esacshiftdone
OLDDIR="
pwd" cd "dirname $0" HOBBITCLIENTHOME="pwd" cd "$OLDDIR"MACHINE="
echo $MACHINEDOTS | sed -e's/\./,/g'"export MACHINE MACHINEDOTS BBOSTYPE BBOSSCRIPT HOBBITCLIENTHOME
case "$CMD" in "start") if test ! -w $HOBBITCLIENTHOME/logs; then echo "Cannot write to the $HOBBITCLIENTHOME/logs directory" exit 1 fi if test ! -w $HOBBITCLIENTHOME/tmp; then echo "Cannot write to the $HOBBITCLIENTHOME/tmp directory" exit 1 fi
if test -f $HOBBITCLIENTHOME/logs/clientlaunch.`hostname`.pid; then echo "Hobbit client already running, re-starting it" $0 stop rm -f $HOBBITCLIENTHOME/logs/clientlaunch.`hostname`.pid fi $HOBBITCLIENTHOME/bin/hobbitlaunch --config=$HOBBITCLIENTHOME/etc/clientlaunch.cfg --log=$HOBBITCLIENTHOME/logs/clientlaunch.log --pidfile=$HOBBITCLIENTHOME/logs/clientlaunch.`hostname`.pid if test $? -eq 0; then echo "Hobbit client for $BBOSTYPE started on $MACHINEDOTS" else echo "Hobbit client startup failed" fi ;;"stop") if test -f $HOBBITCLIENTHOME/logs/clientlaunch.
hostname.pid; then killcat $HOBBITCLIENTHOME/logs/clientlaunch.$MACHINEDOTS.pidecho "Hobbit client stopped" else echo "Hobbit client not running" fi ;;"restart") if test -f $HOBBITCLIENTHOME/logs/clientlaunch.
hostname.pid; then $0 stop else echo "Hobbit client not running, continuing to start it" fi$0 start ;;esac
exit 0
-----Original Message----- From: David Gore [mailto:David.Gore at mci.com] Sent: Wednesday, January 11, 2006 9:52 AM To: hobbit at hswn.dk Subject: Re: [hobbit] multiple hosts
Rolf,
I asked a similar question many months ago for our Digital UNIX (4.x) boxes running in multiple clusters. I did not get any answer, so I had to create a hack that does run multiple clients on each host. One client is for the physical host and does, cpu, disk, etc, and one client for the logical/virtual host that does the tasks specific to the logical/virtual host. The runclient.sh script runs every ten minutes via crontab, to make sure both clients are running and starts either client if they are not. I also have custom versions of some of the default Hobbit client scripts to support multiple clients on a single host. So in short as far as I know, you are going to have to hack/script your own solution. I don't mind sending you what my hobbit scripts have turned into, but it may not be what you are looking for.
David Gore
*e-mail via SUSE Linux 9.3 and other open source tools.
Rolf Schrittenlocher wrote:
Hi,
as there was no answer to my first mail, I try again with other words:
Our machines may have one or more aliases (virtual hosts) which all have DNS-entries. Normally we refer to the machines not using the hostname but by the alias, each of them combined to a bundle of functions and tasks. So a machine "frodo.xxx.de" might host "webserver.xxx.de", "dnsserver.xxx.de" and "tolkserver.xxx.de". These aliases and of course the tasks related to them are transferred to another machine in case of hardware failure, maintenance, etc. In bb-hosts we have an entry for each machine (to get data from this machine in case there is no alias set on it) and of course for all relevant aliases. What we need now is that the general client data (cpu, disk, ssh, ...) for each machine appears both with the hostname and all aliases set on it. I added a script to runclient.sh which creates ~/client/etc/clientlaunch.cfg dynamically according to the aliases set on the machine before starting the client. This assures that the correct data is polled on the client. I may modify $MACHINE or Í may use the CLIENT:hostname tag in bb-hosts to make the client send messages with a different name. But that would require several instances of the client running, somethin I dislike. I hope there is a way to send the client data once and to make the server look at some configuration file to see where he should present these data. Maybe that possibility exists but I haven't found it yet. So I hope that someone could indicate me where to search for the places which I have to "hack" hoping that it is possible to do that with shell or perl as my C-knowledge isn't so good. But even if C is necessary: Any hints are appreciated.
Thanks Rolf
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
participants (6)
-
adam.scheblein＠marquette.edu
-
astavitsky＠yahoo.com
-
fabio.flores＠flytxt.com
-
henrik＠hswn.dk
-
lebarber＠gmail.com
-
rmunsch＠solutionsforprogress.com