brokebit’s

 http://apple2history.org/history/ah07.html

It was a crazy weekend. My CallManager/BGP network upgrades got canceled because we were having connectivity problems to remote sites and we didn't want to introduce any more changes until all the problems got resolved. We started getting a bunch of complaints that users could use email and that messages would just sit in users outboxes. Outlook would eventually just pop up one of its "Trying to connect" messages balloons down by the task bar. We proceeded to troubleshoot the issue to our San Francisco office and New York offices and discovered that any packet with a bytes size larger the 128 bytes was getting dropped. Both of our office are in UBS office buildings and UBS provides all of the network equipment and connectivity. So trouble shooting from their end was very difficult. Also both sites are connected back to headquarters via a IPSEC connection. In the end we discovered that if we installed the Cisco VPN client on the end users machines and used that to connect instead of the L2L tunnel everything worked fine. So today we are chasing after UBS to try and trouble shoot their VPN concentrator which terminates the remote end of the L2L tunnel.

 Also on Saturday we were installing new A/C units in our data center and relocating network equipment to make room for two new 6500 core switches. During that process many of the machines in the data center lost connectivity for short periods of time. After the move was completed Nagios started screaming that half the network was down. I immediately thought this was a problem with Nagios or the Nagios box itself because everything else seemed fine. I spent about four hours playing with duplex settings, speed settings, kernel drivers, etc on the Nagios box to try and figure out what the problem was. Then I started to notice that I was seeing issues outside of Nagios. If we attempted to ping any host within our production subnet we experienced about 40% - 80% packet loss. Nagios wasn't't broke! It was doing exactly what it was supposed to do! It was telling me the network was f&8^ked up! Long story short — we narrowed the problem down to a single bad SFP GBIC in one of the IDF closets.

Product Features

Technical Details

• Screen size: 32 inches
• Color: Black
• TV type: LCD
• HDTV capable: Yes, with integrated HDTV/ATSC tuner
• Game mode: Yes
• Resolution: 1366 x 768 pixels
• Aspect ratio: 16:9
• Speaker wattage per channel: 10 watts
• Number of speakers: 2
• Response time: 8 ms
• Brightness: 500 cd/m2
• Contrast ratio: 4000:1
• Viewing angle: 178 degrees
• Comb filter: Yes, 3D Y/C digital
• 3:2 pulldown correction: No
• Scan velocity modulation: No
• Line doubling: No
• Color temperature control: No
• Includes DVD player: No
• Includes VCR: No
• Signal type: NTSC, ATSC
• Auto channel setup: Yes
• Closed captioning: Yes
• Last-channel recall: Yes
• Multilingual menu: Yes (English, French, Spanish)
• Parental control: Yes
• Picture-in-picture (PIP): Yes
• Remote control: Yes
• SAP: Yes
• Sleep timer: Yes
• Composite A/V: 2 ( side)
• S-Video: 2 ( side)
• Component video: 1
• DVI: 0
• HDMI: 1
• RF: 2
• PC VGA: 1
• USB: 1
• Firewire: 0
• RS232 (Serial): 1
• Headphone jack: 1
• Front/Side AV jacks: Side
• Surround sound: Yes, virtual SRS TruSurround XT
• Sound leveler system: Yes
• Item width with stand: 31.5 inches
• Item height with stand: 23.7 inches
• Item depth with stand: 9.9 inches
• Item width with stand: 31.5 inches
• Item height with stand: 21.6 inches
• Item depth with stand: 3.1 inches
• Item weight: 32.6 pounds
• Stand included: Yes
• Wall mount type: VESA

No it wasn't. I moved the server over to the T1 line a couple of days ago and Easydns didn't reload my zone files. So my primary namerver had the correct IP and the secondary nameservers had the old IP. I had to force the Easydns servers to perform a reload of the zones. What kind of crap is that!

Anyone interestead in some co-locaiton space? Know anyone that is? I'v got cheap bandwidth available. 

Scenario:

I have a routed ADSL circuit with 32 static IP address connected to a Cisco 2600 via an ADSL WIC. I also have a T1 line with 32 static addresses connected to Cisco 3620 via a T1 WIC. (Both of these connections are through Speakeasy.) I wanted to provide redundant internet connections to my hosts without having to use BGP.

The following configurations will allow the two routed subnets to communicated with one another via Ethernet as opposed to going out over one line and coming back in on the other. They will allow each subnet to fail over out-bound connectivity to the other link in the event one goes down via NAT. You can use this setup to offer some redundant inbound mail routing. These configurations can be extended to any type of connection and don't necassarlily need to be for a T1 line and ADSL.

These configurations also assume your routers have a single physical Ethernet interface and you are using VLANs and dot1q trunks. I have put comments in red so please don't paste those into your router.

 
Config #1 (T1 Router):

ip domain name brokenbytes.org
ip dhcp excluded-address 192.168.100.1 192.168.100.229
ip dhcp excluded-address 192.168.100.240 192.168.100.254

I wanted to run DHCP for my internal network. You don't need this if you don't want it.

ip dhcp pool home   
   network 192.168.100.0 255.255.255.0   
   default-router 192.168.100.2
   domain-name brokenbytes.org
   dns-server 192.168.100.1

interface FastEthernet1/0 
 description internal interface  no ip address
 speed 100
 full-duplex

This is my internal network interface. (VLAN 4). Users client PCs are on this subnet and NATed out to the Internet. This interface is also used to manage the router.

interface FastEthernet1/0.4 
 description management and internal interface 
 encapsulation dot1Q 4 native 
 ip address 192.168.100.2 255.255.255.0 
 ip nat inside

Hosts that need public static IPs are in this VLAN (6).
interface FastEthernet1/0.6  
  description Routed T1 Subnet 66.253.38.192/255.255.255.224 
 encapsulation dot1Q 6 
 ip address 66.253.38.193 255.255.255.224

The only two things on this subnet are the two routers. This subnet is used to route between the two public address spaces on each line without going out over the Internet and to allow us to do NAT from one public range to the other line.

interface FastEthernet1/0.11 
 description Inter-Router VLAN 
 encapsulation dot1Q 11 
 ip address 10.1.1.2 255.255.255.0 
 ip nat inside

interface Serial1/0
 description T1 to Speakeasy Circuit ID
 mtu 1522
 bandwidth 1536
 ip address 66.92.134.92 255.255.255.0
 ip nat outside
 encapsulation frame-relay IETF
 no fair-queue
 frame-relay map ip 66.92.134.1 16 broadcast IETF (Speakeasy requires this..)

This tells the router what to nat.
ip nat inside source list 1 interface Serial1/0 overload

These next two line are what give us the redundant routing to the Internet. Our primary default gateway is 66.92.134.1(Serial1/0). But we have another default gateway with a higher metric. If Serial1/0 is down all traffic will be routed to 10.1.1.1 which is the ADSL router.
ip route 0.0.0.0 0.0.0.0 66.92.134.1
ip route 0.0.0.0 0.0.0.0 10.1.1.1 110

This static route tells the router it should send traffic destined for the 32 DSL public IPs via our inter-vlan subnet (VLAN 11, 10.1.1.0/24) instead of via the Internet.
ip route 69.17.27.192 255.255.255.224 10.1.1.1

These two networks should get NATed when going out outside interfaces. (Serial1/0) This is what allows the DSL subnet to get to the Internet via the T1 line when the DSL line is down. It also allows our Internal network to get NATed out the T1.
access-list 1 permit 192.168.100.0 0.0.0.255
access-list 1 permit 69.17.27.192 0.0.0.31

Config #2 (ADSL Router):

interface ATM0/0
 ip address 64.81.146.241 255.255.255.0
 ip nat outside
 no ip mroute-cache
 no atm ilmi-keepalive
 dsl operating-mode auto
 pvc 0/35
 encapsulation aal5snap
 

interface Ethernet0/0
 no ip address
 half-duplex

Hosts that need public static IPs are in this VLAN (6).
interface Ethernet0/0.9
 encapsulation dot1Q 9
 ip address 69.17.27.193 255.255.255.224
 ip nat outside

Once again, this the same as above. This subnet is used to route between the two public address spaces on each line without going out over the Internet and to allow us to do NAT from one public range to the other line.
interface Ethernet0/0.11
 description inter router vlan
 encapsulation dot1Q 11
 ip address 10.1.1.1 255.255.255.0
 ip nat inside

Nat everything going outside that sourced from either the internal network or the T1 public range.
ip nat inside source list 1 interface ATM0/0 overload

Here is some of our redundant routing. Our primary default route is 64.81.146.1 which is over the DSL line.
If you can't get out that way then send it on over to the T1 router. Finaly we need to add some static routes for
the internal network and public T1 range that point to the other router.

ip route 0.0.0.0 0.0.0.0 64.81.146.1
ip route 0.0.0.0 0.0.0.0 10.1.1.2 110
ip route 66.253.38.192 255.255.255.224 10.1.1.2
ip route 192.168.100.0 255.255.255.0 10.1.1.2

These define what should be nated. basicly stuff sourced from the internal network and the the public T1 Ip space.

access-list 1 permit 192.168.100.0 0.0.0.255
access-list 1 permit 66.253.38.192 0.0.0.31

The holiday is over. It was a long three days of grilling, drinking, bouncing on the trampoline, swimming, and just plain having fun. I’m exhausted! This is the first down time I have had.

Here the quick updates with photos.

1) T1 line got installed today. On Memorial day! Covad works on Memorial day! More to follow on this later.

2) Here is a photo of the WiFi antenna installed.

3) All the plants and veggies have been moved into the ground!

My brother is obssessed with disk arrays. Here is his latest project. A 40 disk fiber channel array!

Yesterday I spent the day setting up SMS paging from Nagios. In the past I had just used email to SMS gateways for sending notifications to my cell. SMS Gateways rely on the Nagios host having access to them and the network infrastructure in between them to be functional. Unfortunately many of the times Nagios needs to send out notifications the network is not in a reliable state. I have experienced several instances where a major router/switch goes down and Nagios has no way to let me know. Solution: SMS TAP dial-up gateway.

Most cellular service providers provide dial up SMS TAP gateways (Some even offer toll free numbers). These gateways allow you to send SMS messages to cellular devices by using your modem. With this setup Nagios could be completely disconnected from an IP network and still be able to get notifications sent out via a modem and some paging software.

Here is a great source of TAP dial-up numbers for most providers.

http://www.notepage.net/tap-phone-numbers-c.htm

Before you can configure Nagios to use a SMS TAP gateway you need to install some software that actually makes the call and speaks TAP. I decided to use quickpage (http://www.qpage.org/) because it was small, easy to build, and easy to configure. Just download quickpage, untar, run configure followed by a make and make install. (You may also want to take a look at sendmpage. http://www.sendpage.org)
Quickpage operates in a client server manner. A daemon sits and listens for a quickpage client to connect and tell it to send a message. (The qpage binary is both the daemon and cleint depending on which switches are specified.) Before you can start the qpage daemon you need to create an initial configuration file for quickpage. The configuration file sets some of the following options: Which serial port you modem is on, cellular service provider definitions, and recipient pager definitions. You can place the config file in any directory as long as you use the -C switch to tell qpage where it is. I think the default place it will look is /etc/qpage.cf

Here is an example qpage.cf file:

——————————————————————–
#Administrators email
administrator=protect.the@innocent.com

#Make sure qpage can write to this directory. If you start qpage as root
#it will become the daemon account.
queuedir=/var/spool/qpage

identtimeout=5
snpptimeout=60

#Serial port your modem is on
modem=ttya device=/dev/ttyS0

#A service definition called default
service=default
device=ttya
baudrate=1200
parity=even
allowpid=yes
maxtries=6
phone=18886561727

#A service definition called cingular - This seems to work for cingular cell phones
service=cingular
device=ttya
baudrate=1200
parity=even
allowpid=yes
maxtries=6
phone=18668837243

#A service definition called CingularBB - Blackberries
service=cingularBB
device=ttya
baudrate=9600
parity=even
allowpid=yes
maxtries=6
phone=18009094602

#These are pager definitions, Obviously you should replace 5555551212
#with your own cellular number.
#The service tag associates the pager with the services defined above.
pager=eric
pagerid=5555551212
service=default

pager=EdCingular
pagerid=5555551212
service=cingular

pager=EricCingular
pagerid=5555551212
service=cingularBB
——————————————————————–

Save this file somewhere and then execute the following command to start quickpage:

qpage -C /usr/local/etc/qpage.cf -q 5

The will start quickpage and tell it check the queue every five seconds. You may also want to consider adding the -d switch. This will force qpage into debug mode and is very helpful when testing new configurations.

Also note, that some providers like to have a 1 in front of the area code on the pager ID. This was the case with the Cingular dial-up. i.e. 15555551212 instead of 5555551212. They key here is to play around.

Once qpage is running try to send yourself a test SMS message. Quickpage by default will attempt to connect to the qpage daemon running on localhost. This is fine because we are testing from the same machine.
Type: qpage -p eric

Where eric is the name of the pager definition in your qpage.cf. qpage will connect to the daemon and submit a message for delivery. Now watch your qpage debug output. You should see it attempting to dial-out using the modem and connect to the provider. Thats it!

Because quckpage is a client server applications you can actualy run qpage from any host that has IP access to the machine running the qpage daemon. Just use the -s switch and specify the hostname. When thinking about this options it’s important to remember why we are doing this in the firstplace…
Now configure nagios to use qpage:

First you need to define a notification command. I have the following in my misccommands.cfg file:

# notify via sms using qpage
define command{
command_name notify-by-sms
command_line /usr/local/bin/qpage -s localhost -P $CONTACTNAME$ -f $HOSTNAME$ $SERVICEDESC$ ‘$SERVICEOUTPUT$’ $HOSTNAME$
}

Now define a contact that uses the notify-by-sms notification command:

define contact{
contact_name EricCingular
alias Eric’s Blackberry
service_notification_period 24×7
host_notification_period 24×7
service_notification_options w,u,c,r
host_notification_options d,u,r
service_notification_commands notify-by-sms
host_notification_commands notify-by-sms
email top.secret@neutronstar.com
}

Now any service or host that is setup to send notifications to EricCingular will use the notify-by-sms command which calls qpage.

Thats it!

I am a Debian fan. I’ve loved and used Debian since the early days of version 1.3. If any of you know anything about the Debian release cycle, then you know that version 1.3 was a long long time ago. One of my the main reasons I love Debian so much is apt-get. Apt-get is a nice little utility that will fetch deb packages automatically from any source you specify, mainly the official Debian mirrors. To install a package you simply type apt-get install packageName and apt-get does the rest. Debian had apt-get before any other distribution had anything like it. (yum, up2date, pkgget, etc). I love nothing more then doing a apt-get dist-upgrade to move to the new version of Debian relatively painlessly. Oddly enough, I have never really though about making my deb packages. I’m not a professional software developer, so why would I need to make packages?

Over the last month at my new job I have become intimately familiar with building RPM packages of my companies custom trading software. We have many RedHat systems and need to make sure we have the same version of the software installed on all machines. We need to be able to easily remove the software, or upgrade it. I have compiled a lot of software on my Debian machines (Apache, PHP, image libraries, etc) because often times the Debian packages are several versions behind. BUT when it came time to upgrade or remove something that was installed via a make install, forget it! You have no idea what files got dropped on your system. So, I decided to build RPMs of everything we may be installing outside of the stock RedHat RPMS.

My purpose here is give you some quick instructions on how you can build deb packages of Apache or PHP or whatever software you compile on a regular basis so that you can easily remove it.

Build your software as you normally would. (./configure, make)

Now use the DESTDIR option when doing a make install to set an alternate root for your software to be installed to. Typically the default DESTDIR is / but we want all the files that will be installed dropped in a temp directory that we can use for our deb package. So instead of doing a make install do a make DESTDIR=/tmp/apache install

If you look in /tmp/apache you will now probably see something like /tmp/apache/usr/local/apache etc etc etc.

Next we need to create the Debian package control file. Create a directory called DEBIAN in /tmp/apache. (/tmp/apache/DEBIAN)

Now create a file in /tmp/apache/DEBIAN called control. Here is an example control file that I used for an apache build:

———————————————————————
Package: Apache-Custom
Version: 2.2
Section: My_Custom_Packages
Priority: optional
Architecture: all
Essential: no
Depends:
Pre-Depends:
Recommends:
Suggests:
Installed-Size: 1024
Maintainer: Eric Dynowski [eric@dynowski.com
Conflicts:
Replaces:
Provides: httpd
Description: Custom Apache 2.2 installation
.
More descriptive text.
———————————————————————

Next type dpkg -b /tmp/apache /tmp/apache.deb

If everything went right you should have a Debian package of your apache compile! When it comes time to upgrade you can just run dpkg –purge Apache-Custom and everything gets removed from your system.

There is much more to Debian packages and if you really interested I recommend taking a look at the Chapter 6 of the Debian FAQ.

http://www.debian.org/doc/FAQ/ch-pkg_basics.en.html