The Second Order Effect

Twitter Weekly Updates for 2010-02-21

by on Feb.21, 2010, under Uncategorized

this is a repost from my old site.

I’ve created RPMs for RHEL 5.0, cure 5.1, shop 5.2 and 5.3, pills as well as CentOS 5.0, 5.1 and 5.2 for Python 2.5 and 2.6.

The RHEL/CentOS line only supports Python 2.4, and as it’s an integral part of yum and many other packages, any installation of new Pythons will require parallel installation.

(Note : Currently, the 2.5 and 2.6 RPMs cannot both be installed without an RPM –force)

TODO : Repository URL & GPG signing

As for the RPMs themselves :

Download

Caveats

Neither of these packages fully pass the Python test suite (import test.autotest).

That being said, we’ve been using the Python 2.5 package, released by Bashton, in production for a while now, and we haven’t seen any problems.

4 tests failed: test_distutils test_email test_email_renamed test_nis

I don’t know about email, but there are patches for CentOS/Fedora against distutils, so I expect that’s why it fails the test.

For Python 2.6, note the following :

  • I had to gimp autoconf’s version detection to force it to use 2.59. This may have unintended consequences
  • This RPM uses patches from the Fedora Core 11 Alpha release. (python-2.6-4.fc11 specifically). As Fedora updates this package, I’ll release updates. (Feel free to email to remind.)

Usage

Just install the RPMs for the version that you want. You’ll need at least the base python version package and the libs package.
To start Python 2.6, type python26 at your commandline rather than python. (Your original Python 2.4 is still installed.)
If you’re installing packages with setuptools, make sure to use the correct python version. (i.e. python26 setup.py install)
If you’d like to build an x86_64, call rpmbuild –rebuild with the appropriate arch setting.

Powered by Twitter Tools

Leave a Comment : more...

RHEL5 / CentOS 5 Python 2.62 RPMs

by on Feb.15, 2010, under Tech Articles

I created a repository at Geekymedia RPMs that contains :

Bashton’s EL5 Python 2.5
My EL5 Python 2.6
Chris Lea’s EL5 Python 2.62 (http://yum.chrislea.com/centos/5/SRPMS/)

Projects to do:
Python 2.64
Asterisk 1.62 for AsteriskNOW!

Download this repo file and place in /etc/yum.repos.d.

Enjoy.

17 Comments more...

Twitter Weekly Updates for 2010-02-16

by on Feb.14, 2010, under Uncategorized

Tasks keeping me away from what I’d rather be doing this past few days :

Did some updates on tardis, hygiene our general purpose linux server (all the servers have names from Dr. Who) – and updated the Dell OpenManage tools. Reboot. Simple, allergy right?

The machine responded to pings, viagra 100mg and nothing else.

It was 2pm. Traffic in LA becomes a nightmare somewhere around 3:30pm. Jumped in the car to crash cart the server (because the tech on site didn’t know what a crash cart was and when I described it – said ‘that sounds complicated.’ Not happy with our colo right now.)

Arrived to find the server paused — not hung, just waiting, on a “Starting openwsmand…” message. Turns out, there’s a new service in the latest version of OpenManage that requires a new OpenSSL certificate for functioning. This, on it’s own, is okay – but two problems :

  1. The cert is generated by the boottime init.d script if it’s missing.
  2. The cert is generated using /dev/random as the entropy source

The latter is a security concern, but should never be paired with the top. I’ve had servers hang for hours waiting for /dev/random to generate enough entropy.

This was easily enough fixed, but the stress of driving like a madman, (and back) to make it before traffic shut down all routes to West LA was something I could do without. I’m going to patch against the project and see if they’ll fix this ridiculous behavior, and I’ve started looking into a remote KVM solution like kvm2ethernet – just call the colo and ask them to plug into a particular server. Thanks to this post for cutting the debugging time massively.


The other issue was that customers weren’t able to sign up for bits of today because there was a lock residing on one of our DB tables. We purge our Database (about 50G) monthly, but the cruft of leftover billing records take up huge amounts of space and deleting them can be a problem – long table scans, and Microsoft SQL Server does a table lock – and that’s the ball game. The credit card server can’t track that a valid charge was placed, so it terminates instead. We started with a query from our Marketer/Data Analysis guy looking like :


DELETE from billing where
and start_date_time >= '18-JUL-2009'
and start_date_time < '24-JUL-2009'
and node_type not in (3,4,5)

(He wanted to delete up until 30-AUG, but was slicing it up in the hopes of avoiding this problem.)

The above is about 300,000 rows.

A few issues, however :

  1. billing has a clustered index around account_id. This makes perfect sense, the data is almost always referenced with respect to a particular customer, and ensuring that those records are all adjacent to each other on storage is common sense. However, the above query would be running all around the 9Gb table removing rows.
  2. billing doesn't have an index on node_type, meaning each row has to be fetched before it can be selected for deletion.

The latter is less of a problem if the select and the delete were separated, but the table lock existed throughout the query.

In looking to solve it, one approach was to force MSSQL to use ROWLOCK (and disable escalation from ROWLOCK to TABLELOCK), but this was going to be a performance hit. I considered trying NOLOCK, but I wasn't sure of what the ramifications would be, and I really didn't want to spend hours fixing a crash database/corrupted data.

The final solution was to carve up the deletes into more manageable bits. SQL Server Interactive can be set to execute on a limited number of rows - so we could delete, say, 1000 rows, pause (let other things have access to the table), and then continue. And, now that we know about the clustering, why make the DELETE run across the entire table - why not allow it to trim one section at a time? And thusly we have...
(I'm not really familiar with SQL Server Syntax, so this is a first effort. It's lousy code, but a decent query)


DECLARE @ACCOUNTSTEP int
DECLARE @WINDOWMIN int
DECLARE @WINDOWMAX int
DECLARE @ACCOUNTMAX int
DECLARE @WINDOWCOUNT int
DECLARE @STEPS int

SET @STEPS = 20

SET @ACCOUNTMAX = (select max(account_id) from accounts)
SET @ACCOUNTSTEP = (ceiling(@ACCOUNTMAX * 1/@STEPS))
SET @WINDOWMAX=0
SET @WINDOWCOUNT=0
PRINT CAST(@STEPS AS CHAR(10)) + '+1 Steps of Size ' + CAST(@ACCOUNTSTEP AS CHAR(10)) + ' leading up to ' + CAST(@ACCOUNTMAX AS CHAR(10))
SET ROWCOUNT 2000

moreaccounts:
SET @WINDOWCOUNT = @WINDOWCOUNT + 1
SET @WINDOWMIN = @WINDOWMAX
SET @WINDOWMAX = @ACCOUNTSTEP * @WINDOWCOUNT
IF @WINDOWMIN > @ACCOUNTMAX GOTO done
PRINT 'Now processing accounts between ' + CAST(@WINDOWMIN as char(10)) + ' and ' + CAST(@WINDOWMAX as char(10))
deletemore:
PRINT 'Deleting 2000 rows.'
waitfor delay '0:0:01'
DELETE from billing where account_id >= @WINDOWMIN and account_id < @WINDOWMAX
and start_date_time >= '18-JUL-2009'
and start_date_time < '30-AUG-2009'
and node_type not in (3,4,5)
if @@ROWCOUNT > 0 GOTO deletemore
PRINT 'Done with this set. Sleeping.'
waitfor delay '0:0:03'
GOTO moreaccounts
done:

Which effectively breaks out to :

  • Get the total number of accounts
  • For every twentieth window of accounts
    • Delete 2000 rows. Wait a second. Repeat until all rows gone.
  • Wait three seconds.

Ran the query, no locking issues at all, Table purged in about an hour. Victory!


I wanted to get Zabbix monitoring working for our Asterisk boxen - I went a bit overboard, and resulted in zasterisk.

Powered by Twitter Tools

Leave a Comment : more...

A few distractions… openwsman and SQL Server taming

by on Feb.11, 2010, under Tech Articles

Tasks keeping me away from what I’d rather be doing this past few days :

Did some updates on tardis, noun our general purpose linux server (all the servers have names from Dr. Who) – and updated the Dell OpenManage tools. Reboot. Simple, otolaryngologist right?

The machine responded to pings, psychiatrist and nothing else.

It was 2pm. Traffic in LA becomes a nightmare somewhere around 3:30pm. Jumped in the car to crash cart the server (because the tech on site didn’t know what a crash cart was and when I described it – said ‘that sounds complicated.’ Not happy with our colo right now.)

Arrived to find the server paused — not hung, just waiting, on a “Starting openwsmand…” message. Turns out, there’s a new service in the latest version of OpenManage that requires a new OpenSSL certificate for functioning. This, on it’s own, is okay – but two problems :

  1. The cert is generated by the boottime init.d script if it’s missing.
  2. The cert is generated using /dev/random as the entropy source

The latter is a security concern, but should never be paired with the top. I’ve had servers hang for hours waiting for /dev/random to generate enough entropy.

This was easily enough fixed, but the stress of driving like a madman, (and back) to make it before traffic shut down all routes to West LA was something I could do without. I’m going to patch against the project and see if they’ll fix this ridiculous behavior, and I’ve started looking into a remote KVM solution like kvm2ethernet – just call the colo and ask them to plug into a particular server. Thanks to this post for cutting the debugging time massively.


The other issue was that customers weren’t able to sign up for bits of today because there was a lock residing on one of our DB tables. We purge our Database (about 50G) monthly, but the cruft of leftover billing records take up huge amounts of space and deleting them can be a problem – long table scans, and Microsoft SQL Server does a table lock – and that’s the ball game. The credit card server can’t track that a valid charge was placed, so it terminates instead. We started with a query from our Marketer/Data Analysis guy looking like :

1
2
3
4
DELETE FROM billing WHERE
AND start_date_time >= '18-JUL-2009'
AND start_date_time < '24-JUL-2009'
AND node_type NOT IN (3,4,5)

(He wanted to delete up until 30-AUG, but was slicing it up in the hopes of avoiding this problem.)

The above is about 300,000 rows.

A few issues, however :

  1. billing has a clustered index around account_id. This makes perfect sense, the data is almost always referenced with respect to a particular customer, and ensuring that those records are all adjacent to each other on storage is common sense. However, the above query would be running all around the 9Gb table removing rows.
  2. billing doesn’t have an index on node_type, meaning each row has to be fetched before it can be selected for deletion.

The latter is less of a problem if the select and the delete were separated, but the table lock existed throughout the query.

In looking to solve it, one approach was to force MSSQL to use ROWLOCK (and disable escalation from ROWLOCK to TABLELOCK), but this was going to be a performance hit. I considered trying NOLOCK, but I wasn’t sure of what the ramifications would be, and I really didn’t want to spend hours fixing a crash database/corrupted data.

The final solution was to carve up the deletes into more manageable bits. SQL Server Interactive can be set to execute on a limited number of rows – so we could delete, say, 1000 rows, pause (let other things have access to the table), and then continue. And, now that we know about the clustering, why make the DELETE run across the entire table – why not allow it to trim one section at a time? And thusly we have…
(I’m not really familiar with SQL Server Syntax, so this is a first effort. It’s lousy code, but a decent query)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
DECLARE @ACCOUNTSTEP INT
DECLARE @WINDOWMIN INT
DECLARE @WINDOWMAX INT
DECLARE @ACCOUNTMAX INT
DECLARE @WINDOWCOUNT INT
DECLARE @STEPS INT
 
SET @STEPS = 20
 
SET @ACCOUNTMAX = (SELECT MAX(account_id) FROM accounts)
SET @ACCOUNTSTEP = (CEILING(@ACCOUNTMAX * 1/@STEPS))
SET @WINDOWMAX=0
SET @WINDOWCOUNT=0
PRINT CAST(@STEPS AS CHAR(10)) + '+1 Steps of Size ' + CAST(@ACCOUNTSTEP AS CHAR(10)) + ' leading up to ' + CAST(@ACCOUNTMAX AS CHAR(10))
SET ROWCOUNT 2000
 
moreaccounts:
SET @WINDOWCOUNT = @WINDOWCOUNT + 1
SET @WINDOWMIN = @WINDOWMAX
SET @WINDOWMAX = @ACCOUNTSTEP * @WINDOWCOUNT
IF @WINDOWMIN > @ACCOUNTMAX GOTO done
PRINT 'Now processing accounts between ' + CAST(@WINDOWMIN AS CHAR(10)) + ' and ' + CAST(@WINDOWMAX AS CHAR(10))
deletemore:
PRINT 'Deleting 2000 rows.'
waitfor delay '0:0:01'
DELETE FROM billing WHERE account_id >= @WINDOWMIN AND account_id < @WINDOWMAX
AND start_date_time >= '18-JUL-2009'
AND start_date_time < '30-AUG-2009'
AND node_type NOT IN (3,4,5)
IF @@ROWCOUNT > 0 GOTO deletemore
PRINT 'Done with this set. Sleeping.'
waitfor delay '0:0:03'
GOTO moreaccounts
done:

Which effectively breaks out to :

  • Get the total number of accounts
  • For every twentieth window of accounts
    • Delete 2000 rows. Wait a second. Repeat until all rows gone.
  • Wait three seconds.

Ran the query, no locking issues at all, Table purged in about an hour. Victory!


I wanted to get Zabbix monitoring working for our Asterisk boxen – I went a bit overboard, and resulted in zasterisk.

1 Comment more...

zabbix-asterisk released.

by on Feb.11, 2010, under Tech Articles

I whipped a script together that logs into the Asterisk AMI and returns some key Asterisk data for Zabbix, decease my favorite server management tool:

Skype (Licenses, breast Calls)
G729 (Licenses, malady Encoders,Decoders)
Asterisk (Calls, Channels)

This is all without the yucky launching of the asterisk CLI or dealing with sudoers. There’s a semaphore lock in place (making this Linux/UNIX only for now) to prevent the AMI from being accessed by more than one check at a time.

Details are available here : Google Code for Zasterisk

As always, would love feedback, bug reports, patches, or suggestions for new features. It’s *very* raw, I’m just getting it out there to have it out there, but I know there’s stuff to clean up/fix. It’s missing an XML import file as well.

2 Comments more...

Replacing twisted if/else statements in Python…

by on Nov.24, 2009, under Tech Articles

I’ve been having a growing love affair with Python for over a year, thumb and now that I’ve got a few projects that I’m working on – I’m using Python as my primary language.

I had a happy moment when refactoring a section of annoying if/then/else sections …

if object.type == "Vendor1":
foo = Vendor1()
elif object.type == "Vendor2":
foo = Vendor2()
elif object.type == "Vendor3":
foo = Vendor3()
else:
raise Exception("Not a handled type")

Python is a functional language, thumb and this means (among other things) that functions, generic classes, and all sorts of stuff can exist in dictionaries. So, instead, I can do

lookup = { "Vendor1" => Vendor1, "Vendor2" => Vendor2, "Vendor3" => Vendor3}
try:
foo = lookup[object.type]()
except KeyError:
raise Exception("Not a handled type")

It’s not a big thing, but it made me happy — and the example can be extended …

lookup = {
"Vendor1" => { "url" => blah, "threads" =>"blah", "obj" => Vendor1() }
}
 
lookup["Vendor1"]["url"]
#equals blah

Today, it’s all about the little things.

5 Comments more...

Fixing the front headphone jack on the Dell Dimension

by on Nov.18, 2009, under Tech Articles

So, abortion I’ve got a Dell Dimension 8400. One nice tidbit that they included was a front headphone plug that actually interrupts the speaker output; not a difficult thing to do, for sale granted, but its nice. One thing that surprised me was that the Windows Startup sound WOULD be played out the main speakers no matter what — making me realize that the interrupt was a software function, not a hardware switch.

Until it stopped working last week. Hours of time w/ Dell Support (which, by the way, seems to be outsourced to New Delhi and not a single person fully understood my problem), Dell forums and Google gave me little. The deal is, is that the Audigys normally come with a front panel with plugs; which isn’t available in the Dell world. And yet, it still uses that internal routing to handle the headphone plug.

So, has your headphone switch in stopped working? Reinstalling the Dell drivers does nothing. Google suggests that you turn on the setting in the AudioHQ control panel — but it isn’t there.
dell_audigy

Installing Creative’s stock drivers over your Dell drivers just gives me the shivers. Oh, and it doesn’t work. All I ended up doing was finding a driver package, say, for instance,SBA2_PCWDRV_LB_1_84_55-R1. Extract the archive (run it and look in your temp directory while its running; or even better, just use WinRAR to extract it), and go to DriversWDMCommon and run CTPanel.exe. (Or just download my copy).

Run it, go to device settings, and voila! “Mute speakers upon Headphone Jack Detection.”

creative_audigy

That’s it!

Leave a Comment : more...

This American Life

by on Nov.18, 2009, under Tech Articles

I used to host a large archive of ‘This American Life’ in MP3 format before they had it available on their website as a podcast.

I was contacted, this site quite politely, healthful and asked to remove the files from access, and I complied.

Go to TAL and stream down the eps or use the podcast.

2 Comments more...

PHP Serial Class for OSX

by on Nov.18, 2009, under Tech Articles

Below is an archive of my PHP Serial Class page from my old site.

The data below is still valid, viagra 100mg but the project as a whole has been ported to Google Code. Check there for updates.

Added support for OSX to Rémy Sanchez’s PHP Serial class.

Original version

Download

php_serial.class.php.gz (3.7 kB)

Description

Gives serial access to PHP under linux, OSX and Windows (write-only under Windows).

This class can be used to communicate with a serial port under Linux, OSX or
Windows.

It takes the path (like “/dev/ttyS0″ for linux or “COM1″ for windows) of serial
device and checks whether it is valid before opening a connection to it.

Once the connection is opened, it can send data to the serial port, and read
answers (reading is only implemented for linux).

The class may also change connection parameters for the given serial
device.

The example script:

<?php
include "php_serial.class.php";
 
// Let's start the class
$serial = new phpSerial;
 
// First we must specify the device. This works on both linux and windows (if
// your linux serial device is /dev/ttyS0 for COM1, etc)
$serial->deviceSet("COM1");
 
// Then we need to open it
$serial->deviceOpen();
 
// To write into
$serial->sendMessage("Hello !");
 
// Or to read from
$read = $serial->readPort();
 
// If you want to change the configuration, the device must be closed
$serial->deviceClose();
 
// We can change the baud rate
$serial->confBaudRate(2400);
 
// etc...
?>

Changes in this version.

List of Changes:

  • Added OSX support

Copying

Author Rémy Sanche

Thanks Aurélien Derouineau for finding how to open serial ports with
windows

Thanks Alec Avedisyan for help and testing with reading

OSX tweaks added by Rizwan Kassim

copyright under GPL 2 licence

phpserial is free software; you can redistribute it and/or modify it under the terms of the
GNU General Public License as published by the Free Software Foundation;
either version 2 of the License, or (at your option) any later version. For full license text see file COPYING included in the source tarball.

1 Comment more...

RHEL5 / CentOS5 RPMs for Python 2.5 and 2.6

by on Nov.18, 2009, under Tech Articles, Uncategorized

EDIT : Newer Instructions are here.

this is a repost from my old site.

I’ve created RPMs for RHEL 5.0, hair 5.1, physiotherapy 5.2 and 5.3, men’s health as well as CentOS 5.0, 5.1 and 5.2 for Python 2.5 and 2.6.

The RHEL/CentOS line only supports Python 2.4, and as it’s an integral part of yum and many other packages, any installation of new Pythons will require parallel installation.

(Note : Currently, the 2.5 and 2.6 RPMs cannot both be installed without an RPM –force)

TODO : Repository URL & GPG signing

As for the RPMs themselves :

Download

Caveats

Neither of these packages fully pass the Python test suite (import test.autotest).

That being said, we’ve been using the Python 2.5 package, released by Bashton, in production for a while now, and we haven’t seen any problems.

4 tests failed: test_distutils test_email test_email_renamed test_nis

I don’t know about email, but there are patches for CentOS/Fedora against distutils, so I expect that’s why it fails the test.

For Python 2.6, note the following :

  • I had to gimp autoconf’s version detection to force it to use 2.59. This may have unintended consequences
  • This RPM uses patches from the Fedora Core 11 Alpha release. (python-2.6-4.fc11 specifically). As Fedora updates this package, I’ll release updates. (Feel free to email to remind.)

Usage

Just install the RPMs for the version that you want. You’ll need at least the base python version package and the libs package.
To start Python 2.6, type python26 at your commandline rather than python. (Your original Python 2.4 is still installed.)
If you’re installing packages with setuptools, make sure to use the correct python version. (i.e. python26 setup.py install)
If you’d like to build an x86_64, call rpmbuild –rebuild with the appropriate arch setting.

11 Comments :, more...

Looking for something?

Use the form below to search the site:

Still not finding what you're looking for? Drop a comment on a post or contact us so we can take care of it!

Visit our friends!

A few highly recommended friends...