brokebit’s

Who should coordinate this?

Byron suggested integrating Nagios with Asterisk and Festival. This would allow Nagios to send out voice notifications via a telephone call.

I think you would need to install a POTS card in the nagios server and not depend on some type of IP connectivity to make the call. Just in the case the network is totally down.

Could be kind of cool..

I was hired to be a Linux administrator. My hiring manager has seen my resume and knows that my skills go beyond just Linux/UNIX. My hiring manager decides to take advantage of those skills today!

We had an IT coordination meeting today to discuss all current projects and ongoing work. We had to determine how much of our time was spent on which tasks, etc, etc.

As of today my responsibilities have expanded, scope has increased, and diversified.

I now support:

1. RedHat Linux Servers
2. NextVu network monitoring system
3. Nagios system and services monitoring system
4. Veritas/Symantec Netbackup for Linux systems
5. Cisco AVVID (CallManager and Unity)
6. Backup LAN/WAN Cisco Admin
7. APC InfrastruXure implementation
8. About 8 Linux developers.

WOW! That was fun. I guess they don’t intend on getting rid of me anytime soon.

The NextVu product is pretty cool. It’s a custom sniffer that runs on a modified version of Fedora. The box has six NIC cards that can be connected to various parts of the network on Cisco span enabled ports. It provides a Java based GUI that you can use to monitor traffic flows on a per-application, protocol, session, host, etc level. Some uses would include: Holding WAN vendors to their SLAs, troubleshooting application network traffic, monitoring utilization, determining which applications are the bandwidth hogs, etc. The closest open source thing I can thing of is a cross between NTOP and ethereal.

I setup samba today with Active Directory integration. This was fairly easy based on the HOWTO document available on the Internet. The difficult part was UID mapping between the AD accounts and local accounts in the passwd file.

This all assumes that you have already configured Kerberos and kinit works for you and that you have winbindd configured properly. You can find instructions on how to this here:

• http://www.samba.org/samba/docs/man/Samba-HOWTO-Collection/winbind.html

Here is the scenario:

• All systems have Kerberos configured on them and pam is setup to use that for authentication.
• When you SSH in you need to have a local account in /etc/passwd, but you are authenticated against the AD via Kerberos.
• All systems have a copy of the same passwd and shadow files so the UIDs are the same across the board.
• Samba is installed and configured to use winbindd for authentication.

The Problem:

When you map a drive via Samba you get authenticated using your AD credentials and all files you create are owned by your AD UID. Remember that the Samba/Winbind does a UID GID mapping and this doesn’t correlate with what’s in /etc/passwd. If all users were only using Samba on all servers this would not be a problem, but we are also using NFS which is using the local passwd file.

Solution:

Remark out the idmap lines in your smb.conf and add username map = /etc/samba/username_map.txt.

Username_map.txt is a text file, which contains a manual mapping of local account names to AD account names. Here is an example:

Unix Account = AD Account
dynowski = domain#dynowski

Now samba will know that domain#dynowski is really dynowski from the /etc/passwd file and use the UID when dealing with permissions and ownership.

Here is an example of my smb.conf

[global]
unix charset = LOCALE
workgroup = DOMAIN
realm = DOMAIN.COM

server string = evprodlx01

# separate domain and username with ‘\’, like DOMAIN\username
winbind separator = \

# use uids from 10000 to 20000 for domain users
#idmap uid = 10000-20000
# use gids from 10000 to 20000 for domain groups
#idmap gid = 10000-20000
# allow enumeration of winbind users and groups
#winbind enum users = yes
#winbind use default domain = Yes
#winbind enum groups = yes

# give winbind users a real shell (only needed if they have telnet access)
template homedir = /home/%U
# template shell = /bin/bash

username map = /etc/samba/username_map.txt
encrypt passwords = yes
security = ADS
password server = 10.1.2.11

[data]
comment = data volume
read only = No
path = /data
guest ok = Yes

[homes]
comment = Home Directories
read only = No
guest ok = Yes

Here is a nice command line capture utility for an iSight or Macbook camera.

Just type isightcapture test.jpg and you get a nice test.jpg from your iSight or Macbook camera. Works great for settings up webcam or any type of scripting.

isightcapture.zip

I found it at: http://www.intergalactic.de/hacks.html

I (sadly) wiped the 64bit AMD machine today and re-installed the 32bit version of RedHat AS 4. It’s amazing how many problems I have run into trying to port various RPMs and libraries to the 64bit AMD platform.

• Many applications still look for the 32bit version of libc and blow up/don’t compile if they can’t find it.
• Some libraries want to install into /usr/lib64 and others into /usr/lib.
• When compiling some apps are platform aware and look in /usr/lib64 while others are not and they look in /usr/lib
• Some commercial libraries are not available in 64bit versions. (I don’t understand why it’s such a big deal for some to type make!)

I was able to hack my way around most of these issue by creating softlinks, or screwing around with the SPEC files to get things in thier proper locations but in the end we simply had to move back to a 32bit OS because of all the various issues raised as a result of the items listed. There were just too many customizations made to keep track of.

This is rather disappointing. I think the developers would really love to be able to take advantage of the 64bit platform, but unless the open source community makes a concerted effort to try and make the shift to 64bits on Linux transparent nobody will attempt to move over, or end up moving back to 32bits after getting tangled in the 64Bit x86 web.

Strange thing is that I don’t recall having any of these problems on other not x86 64bit platforms. I have been running Debian/Redhat on various SPARC, MIPS, and ALPHA platforms of years. So what’s the major difference? Keeping backwards compatibility between 32bit and 64bit x86? Well it isn’t working!

I spent a fair amount of time on Friday configuring Nagios (http://www.nagios.org). I have it monitoring CPU, Memory Usage, System Load, Number of Processes, Services, Disk Usage, etc, etc, etc on both UNIX and Windows hosts. I used the NSClient program on the Windows hosts and ran into a fun little problem when trying to start the service.

First I copied all the appropriate files to c:\program files\NSClient, then I ran pNSclient.exe /install to install it as service. Next I typed net start nsclient and got the following error:

A system error has occurred.

System error 1067 has occurred.

The process terminated unexpectedly.

The event log contained the following:

Event ID: 2 Source: NSClient
NSClient CollectData: Call to rereve counter value for failed, returning stats code 4294967295.

Event ID: 1000 Source: Application Error
Faulting application pNSClient.exe, version 2.0.1.0, faulting module unkown, version 0.0.0.0, fault address 0×008c2459

I am using Windows 2003 with SP1 installed. SP1 tigtned down the security screws a bit and needed a bit more tweaking to allow Nsclient to run a service. The problem is related to the new Data Execution Protection stuff added to SP1. To fix the problem do the following:

Right click My Computer -> Choose Properties
Click on the Advanced Tab
Click the Settings button under the Performance section.
Click on the Data Execution Prevention tab
Click the Turn on DEP for all programs and services excpet those I select radio button
Click the Add button and browse to your copy of pNSclient.exe

Thats it.

I also am using nagios to monitor various Cisco statistics and states. I will post links to them shortly with example configs.

Pros:

• Fast! (Core Duo)
• Nice Camera.
• Rosetta really works well. I can never tell if I am running a PPC based app vs a Intel/Universal one.
• Wireless reception is way better the PPC G4.
• Screen seems brighter and crisper then the G4.
• The power connector.
• Dual booting. (bootcamp)
• Thinner the the G4
• I forget that I am using an Intel based machine. It just doesn’t have that PC feeling.

Cons:

• No PCMCIA Slot. (No EVDO Cards can be used)
• Don’t like the touch pad mouse! (I’m using an external bluetooth mouse)
• Bootcamp: XP Blue screened during the install! I restared the install and it worked the second time around. Go figure.
• Bigger power supply.
• No way to right click in Windows XP without an external mouse.
• No Windows camera driver yet.
• No stable Intel native version of NeoOffice, OpenOffice or MS Office (Though rosetta does a really good job)
• Expensive.

Conclusion: I really like it. I don’t regret getting it. The speed improvment is significant and makes up for the few sluggish apps running via Rosetta.

Qemu and Parallels rock!

Work has been totally fun. I don’t miss being responsible for all of infrastructure. I was a little worried at first. I can spend the entire day concentrating on two or three technical projects and not worry about all the other stuff. I miss the old Fujisawa days, but I don’t miss Astellas for second. If anyone else is thinking about leaving I highly recommend it! That place is diseased.

Also, since a bunch of us are leaving, please fill out this form to determine where we should take you to dinner.

• Dining Plan

I am still messing around with building RPMs. Things got a bit more interesting because we now have some 32bit Intel Xeon machines and some 64bit AMD machines. I had to rebuild all the packages on the AMD machine so that they would be 64bit. I thought it would be as simple as just adding the –target option to rpmbuild but no. Many programs failed to compile because many libraries have been moved to /lib64 instead of /lib. So I spent a better part of the day tweaking more SPEC files to get stuff to build.

I also had a pretty major “O’h Shit!” moment. I accidently hit CTRL ALT DEL on the wrong Redhat box and rebooted it! Mother F#@!IN B%#CH! The machine was our primary NFS server with the main data volume being a LVM volume. After the OS came back I noticed that I could not mount /data. So I did a pvscan. That worked. Then I did a vgscan. That worked. Then I did a vgchange -a y to active the the volume group and that didn’t work. I got a bunch of nasty errors about Invalid arguments and not being able to insert devices into the dtree. /dev/data/data did not exist.
This was bad. All the work I had just spent the last 8 days working was in /data and there was no backup yet. Long story short. It was a bug in the stock Redhat ES server kernel. I upgraded from 2.6.9-5 to 2.6.9-22 and everything just work like it was supposed to. Oh how I long for Debian and my custom kernels! I can’t believe we are paying for this shit.

Here is wierd one.. The IT department at my newCo Install OS on the servers, configures routers and switches, runs the phone system, etc, etc.. but contracts with ADT to come and rack mount new servers! Really!

I dropped the Apple G4 Powerbook off at the Apple store to get the display replaced. It suffered from “white spots” (http://www.apple.com/support/powerbook/displayprogram/) problem. They are going to replace the display for FREE! Yippie. Troy is going to buy it when I get it back.

Everyone should go look at the H20 Plus website (http://www.h20plus.com). Lonnie is doing the advertising. Images and ad copy.

I know of a good third shift operations position if anyone is interested!