When I began looking at possible database schemas for storing NetFlow data in MySQL, I was worried about space. I was working under different assumptions back then (the numbers I had for data in was 1/100th what it is now), and I was thinking that perhaps I could keep the whole database under the 32-bit boundary (for old filesystems, portability, etc.). Now that the 2GB barrier is clearly broken with the sheer amount of data I’m dealing with, I’m not concerned at all with disk space – hard drives are cheap, or so says my boss. What does this mean for the schema? No more CPU-disk tradeoffs in favor of disk space. But does that also mean I can drop additional disk space, and a bit of CPU, for coding convenience, maintainability, extensibility, and ease of use?

In the short-term NetFlow tables, data lives for only a month or two at a time. Any more than that and insert times go down the drain — indices get too large for MySQL to cache adequately, even when there’s only two indices on the whole table. One of those indices is on a SMALLINT field (that’s a 16-bit integer for you non-MySQL types) corresponding to data in a SMALLINT-DATETIME lookup table.

Obviously there’s CPU overhead involved in converting the smallint back into a date, or vice versa, whenever the database is accessed. Furthermore, there’s a code inconvenience/overhead involved. Lastly, but certainly an important consideration, is that the system becomes less scalable: when places like NYU have the resources (such as a system with 32GB of RAM, like some of the Sun systems they’ve got there for network security) to keep a couple years’ worth or NetFlow data in MySQL, the system will break because 16-bit integers only allow for just over 1.8 years’ worth of 15-minute reports. Or consider an institution that wished to have more granular data, suppose 5- or 1-minute reports; not even considering the extensive code changes required to move from 15-minute granularity to some other level of granularity, the system comes crashing down in a mere 45 days. This is unacceptable. Even though my superiors at the workplace currently don’t intend on using on using the system in this way, chances are that their demands will change over the next year, and I want to be able to adapt the system to accommodate for those changes.

So now what I have a motivation for change, the question is on the direction in which to make the change. There are a few good options.

Bump the SMALLINT to a MEDIUMINT
Adding 8 bits would get me somewhere a lot more safe in terms of maximum capacity. But again we have the problem of scalability/maintainability if we want to change the granularity of the system. Also, like before, the system uses different date metrics for different tables, and things get confusing as well as difficult to maintain.

Drop the lookup table and move to DATE and TIME fields
Allright, it sounds a lot like overkill to store 48 bits where I could deal easily with 24. I’ll admit that. My logic is that since storage is no longer an object, things like this are possible. CPU-wise, I’m not sure exactly what the tradeoff will look like numerically; there should be a slight increase (okay, maybe not slight) in time taken to insert into the table because indices are larger, but total query time will be less because the number of queries is lower (no need for an extra lookup query on the date field). An added benefit, though, is that the entire backend could work under the same date metric. The huge plus here is that the system is completely scalable and 100% granularity-agnostic. Half minute, even ten second, reports? Sure, you got it. I can’t fathom needing anything less than one-minute reports, but I’m a stupid programmer/engineer, so I’ll quarter my minimum expectations and settle on 15-second reports as an absolute minimum (after which I would expect the system to just be real-time and not periodical in nature, which would require a restructure of the system anyhow).

I’m leaning towards the latter option above, but am open to suggestions and whatever my benchmarks report. I’ll make the final decision on this hopefully by the end of the day, definitely by the end of tomorrow.

The move is official, then. NIC is to be the production name of the Network Information Center we’ve been working on this semester. With any luck, NIC will hit the UConn Network this summer. Stay tuned for more – including Lina’s first post, coming tonight!

Joe, the Apple representative I spoke with at the Co-Op, seemed fairly confident that they would just replace the drive this time. On the service request that gets sent to the support tehcnician, Joe wrote, “The hard drive refuses to mount. Loud vibrations, akin to the blood curdling wail of a harpee, bring men to their knees.“.

Before doing all the paperwork with Joe, I asked if I might borrow a firewire cable and try to mount the drive on one of the machines there at the Co-Op, so as to perhaps salvage some of the data. It was a no-go. The drive wouldn’t even seek, and it made Disk Utility on the other computer go nuts. My last backup was a while back, too. Serves me right, I suppose. Let’s just say I’ll be e-mailing a few professors about the matter.

While the sounds emitted by the hard disk this time around were a bit worse than last time (there are scratching sounds to account for), I don’t think they’ll actually need to replace the drive. I guess we’ll find out on Thursday. Wish me luck.

When Mum heard the news, she told me to have my friends tie me to a chair and never bring a computer near me for that whole week — I’d surely be going through widthdrawal. Widthdrawal or not, it’s definitely going to be an interesting week. Work, School, Gentoo, it’s all going to Hell this week.

How did it happen, you ask? I don’t know for certain, but I can postulate. What happened, you ask? Now that I know for certain. I woke up last night to my roommate inquiring about the sound coming from the computer(s). It turns out the sound was coming from my laptop. Too sleepy to care, I turned it off and went back to bed. After taking a look at it in the morning, I found that the hard drive sounded like a deisel truck. Now that’s never good. The sound didn’t get any better, but I was able to boot the system (remarkably) and back up my Address Book database as well as my iCal calendars, which were really the only two things that I didn’t have an up-to-the-minute backup of. My ~/Library (preferences, etc.) folder had been backed up a while back, and my Music had been recently mirrored onto Charlie’s Server (via the FTP service).

Looks like everything is going to be fine on my end, in terms of saving all my information. I got what I needed, and was working on creating a convenient .dmg of my music collection (to avoid filename mangling issues between different filesystems) when the hard drive noise got louder and Mac OS X decided to become irresponsive. At that point I gave up.

So basically, what went on is that, several months after dropping the laptop on its side (hey, I tripped, it wasn’t my fault) where the hard drive is, the unit started rattling, presumedly against the side of the case. Rattling at 5000rpm is loud as all hell against an alluminum chassis. Not only was the rattling loud, but it also caused enough vibrations to make disk access impossible.

I called the UConn Co-Op technology guys (I know them well) and filled them in. They said that they can ship it out Monday, and have it back to me by [hopefully] Thursday. Apple support has been excellent in the past, and I do have a subscription to the AppleCare Protection Plan that isn’t even near expiry, so I should be just fine. Meanwhile I’m stuck using the command-line on my server and friends’ computers. Pray for me and my laptop.