Ben

Back to blogging in Chicago

ben — Wed, 18 Aug 2010 18:49:52 +0000

The idea of a public blog can be both frightening and exciting. Worst of all, the decision to write or not write lends itself to Barry Schwartz’ paradox of choice. So, here I cave in and try [again] for the iterative approach: write and see what happens. I had help once again getting to this point via a push from some friends. Thank you. So here we go…starting small…

Yesterday I got a free ticket to the Cubs game + free food. An exchange rep was taking one of our traders out, and I lucked upon an extra ticket. The game was only okay — perhaps because the Cubs are perennial losers. Nevertheless, it was great to be in Wrigleyville with some non-tech guys.

Actually the pampering must seem quite low key to the veteran folks. Even 30 year old traders will often talk about how ski vacations and trips to the Bahamas were regular fare back before regulation set in (earlier for the equities crew). Now not even ten years later, even buying your clients a bison-dog (yes, bison) builds a pretty non-insignificant air of sketch. Still, the rep job seems like a good gig… a sort of finance version of the deal-making consultant: interfacing between client traders and ecn-side infrastructure developers. On a side note, one of my friends in consulting is not even a month into training, and the expense card’s effectively uncapped for <$75 outings. Phenomenal. Sometimes it’s good to be in sales et equivalent.

The irony of the outing, perhaps, is that it seems to me that neither the client trader nor the reps are wholly responsible for the majority of the trading infrastructure and executed trades these days. They know the details better than most, but signal detection, modeling, even routing, etc — are all independent roles now at many places. The “trader” has been relegated to the most latter stages of the front office that places them in contact with the likes of brokers and ECNs. A little bit of overspecialization in the name of efficiency? Perhaps. Certainly, an industry needs generalists for some kinds of innovation. Specialization is a sort of economy of scale, but anyone who has had to deal with a coworkers’ specialist tools knows how much mobility is sacrificed for scalability.

So our qualms and innovative ideas end up having to filter through all the operational overhead before they can even be critiqued much less before they can evolve. Luckily most of this was solved yesterday with some beer and Italian beef. Cheap beer though.

Wriggly field has the great benefit of being a sort of boutique among ball parks. It’s smaller, less industrial, and has all these interesting or trendy buildings surrounding it. A lot of these buildings house rooftop bars that overlook the field. Some of these bars are quite nice, and perfect for a group outing or the like. I heard that the owners of the field wanted to build up the bleachers one time, but since the bleachers would block the rooftops, a legal settlement had to be reached. Not something you’d hear about in Boston or NYC.

Anyway, 3 or so hours later we headed back in a Taxi, weaving through all the 20-something Cubs fans and other partygoers. I have to go to that area more often.

my backup experience

ben — Mon, 15 Mar 2010 17:01:37 +0000

This is how I struggle to backup 1 Terabyte of files. I’d like to hear other people’s solutions.

My files are roughly distributed into three groups: personal, media, and data. Personal includes pictures, documents, etc. Media includes movies and music. Data includes financial data, news data, sql databases, etc. I have some other files such as bluray movies and high resolution market data that I don’t attempt to backup.

Personal files and system files (/) on RAID 1 (approx 200GB). This not only ensures an extra level of backup, but also prevents downtime related to system repair and recovering archived files. I experimented for a while with Raid 5, but I found that the only way it made economical sense was if I had several drives devoted to it. However, for cases where I have several terabytes of data (e.g. big databases), I also demand reasonable write time, which Raid 5 has none of. Then, I experimented with Raid 1 across all the data I had. However, this turned out to be pretty difficult to scale, and not particularly better than the final solution I found. So for now, I have my server’s system files and my personal files in Raid. The others rely on the other levels of redundancy:
All data (including personal) rsynced to a separate machine. I have a mac which doesn’t really do anything except run iTunes and other music related software. An external drive is attached to the mac, and I rsync all 1TB from my server to the mac. I experimented with setups where this rsync server is offsite (specifically, in a different state with my parents). However, that made it really difficult when power outages occurred or when hardware broke. The target drive is 2TB, so it can be scaled. Also this means my primary drives aren’t raided, so they can easily be upgraded to a larger disk.
All data on the mac backed up with Mozy. This took an extraordinary amount of time to do the first backup, but Mozy is by far the cheapest option for the amount of space they provide. It is also relatively bug ridden, painfully slow (weeks to months for backup), and resource intensive. I think it would be difficult to run Mozy on a workstation computer while trying to do other tasks, since it will definitely use up an entire core and also do many times the amount of I/O as the size of the files would suggest.

System backup crons should remember to exclude certain directories like so:

sudo rsync -delete -azvv -e ssh –exclude “/export/” –exclude “/mnt/” –exclude “/proc/” –exclude “/media/” / ben@mac:/Volumes/external/backups/server01

fuse-zip

ben — Tue, 02 Mar 2010 22:42:37 +0000

Very useful tool for mounting zip files on linux. The trouble is, there is no deb for Ubuntu karmic, and so you have to either rebuild it yourself, or do a little apt-hack. To apt-hack, all you have to do is update your /etc/apt/sources.list to the 2010 release: lucid (so in vim :%s/karmic/lucid/g). Now do a sudo apt-get update, but be careful not to upgrade! Now install fuse-zip (or you can download the package), and only a few dependencies should be upgraded. Revert sources.list and you’re all set!

Chinese study technique

ben — Wed, 24 Feb 2010 01:44:43 +0000

Okay. I’m going to blog again now, except from my phone. Everything will therefore be nearly twitter length, and thus managable.

For today, I want to point out how awesome chinesepod.com is. The android app is especially useful, and has provided me with hours of company on long drives. I’ve even learned a couple 成语 that even my legit chinese friends don’t know.

To force myself into using chinesepod, I’ve set a daily cron that emails a days worth of preset assignments. Each Saturday I update a file with the week’s assignments and the date of each. So each morning the assignments end up alongside my other emailed tasks.

Anyway, as always…. add oil!

painting idea

ben — Sat, 02 Jan 2010 16:50:52 +0000

A man and a woman facing each other in argument, shown from the chest up on the side. The man wearing a suit and the woman in casualwear. Rather than seeing their face and cheek, the heads are hollowed out (like in cross-sectional diagrams of machines or the like). Each head is drawn to resemble a different type of room, and in each head is standing a replica of the person itself. So there is a man in a suit standing in the head of the man with a suit. The woman might be, in this case, standing on a soapbox with a protest sign. The man might be standing in a corporate office with his arms crossed. The scene is replicated outside of their heads.

Outside of their heads there can be a scene of protest blurred in the background, with two sides separated left to right. Inside, a lot of detail would exist in the furniture in each head-room. Also each room has a window to the outside, which is not the true outside, but an ideal of the future. We see skyscrapers and city scenery in the man’s window, and a garden with vegetables on the woman’s.

At the front of each head-room is a workstation, command console, etc. The man has before him sets of levers and valves, as if he were commanding a small steam ship. The woman, a desk littered with papers and medals, perhaps.

Anyway that is what I was thinking of in the shower. Someone want to be commissioned to make it?

New Blog

ben — Mon, 16 Nov 2009 02:38:21 +0000

I started a new blog for my [generally strong] opinions, which I have avoided posting here. Ask me for a link.

samba / wfs over ssh

ben — Wed, 14 Oct 2009 21:00:08 +0000

Here’s a useful trick for people who like using windows file share / samba outside of their LAN. This is for a windows client when you don’t want to disable windows file share on the client.

1. Run an SSH server on the samba server. Ok you’re done for the server.

2. On your windows client, create a new network connection using the “Microsoft Loopback Adapter”.

3. Disable “File and Printer Sharing” and enable TCP/IP on the adapter. Copy the subnet/gateway/dns information if you want, and set it to an IP that you don’t use, e.g. 10.0.0.1

4. Startup an ssh client like putty and tunnel 139 on the server to your loopback ip, 10.0.0.1:139

5. Now you can access your samba server through \\10.0.0.1, hooray!

Few more notes on the server

ben — Tue, 08 Sep 2009 02:42:37 +0000

The parts came a few days after my previous post, and I built the machine. Here are a few notes for those who might be doing something similar.

Operating System. I cycled through WinXP 64bit, Vista, Ubuntu Server LTS 8.04, Ubuntu Server 9, and Windows Server 2008 before re-settling on Ubuntu Server 9. Although not thoroughly satisfied by any of those operating systems, the main decision came down to random driver support, os overhead, and raid support. I began with a hardware raid 5 through the intel matrix raid controller. This worked ok, but forced me to move to operating systems to supported it nicely (XP does not do a good job with 64bit drivers and hardware support). Windows based systems were nice, but they did not support the raid controller natively. I had the OS run from a non-redundant drive for a while because of this problem. Ubuntu 8.04 was nice in theory, but it did not support my graphics card. I’m quite satisfied with Ubuntu 9, which I will get into the details of later. I’m running it headless, which is kind of a downside, since I get a warm fuzzy feeling by having a gui even if it’s just to hold multiple terminal windows.

Raid. This was a pain. Raid 5 was wonderful until I realized the the I/O performance would be horrible. This was compounded by the fact that I ran the VMs off the same volume as my data. Thus, swap occupied the same physical drives as movies/data/etc, and that made running the machines while copying files a disaster. Raid 5 also was surprisingly bad at writes even without that. To top it all off, it only made sense to use 3+ drives with Raid 5, so I didn’t have any other drives left for raid. So finally.. I decided to split it up into two raid disks, Raid 1 (1TB) and another Raid 1 (1TB). One houses the operating system and low frequency data (e.g. multimedia), and the other stores data that either I change regularly or is used in any heavy way (e.g. market data).

VMWare Server. I ran workstation for a while when I was on windows. The combination of workstation + windows meant that entire gigs of memory disapeared for no apparent memory. The result was I swiched to VMWare server for Ubuntu. The benfits were: it’s free (really!), it runs headless with a web gui, and it scales memory much better (e.g. it swaps unused segments, which hurts performance marginally and lets you allocate a lot more ram per machine). The disadvantages are that the web gui sucks, especially for things like adding/removing hardware or reconfiguring virtual disks. Despite that, I highly recommend it.

Some things I haven’t figured out: 1. Configuring virtual networks in vmware server seems like a command line affair, which I haven’t gotten to. 2. My UPS is not supported out of the box by Ubuntu, and NUT has been horrifically hard to set up. In windows all I had to do was run an install and forget it. I might have to hook up the UPS to a separate computer and run a shutdown script remotely.

All in all, it has been a mini adventure. Now I’m re-running backups with Mozy, which will probably finish by the end of this week — a full month after I started this project. I’m very happy with how it turned out.

Building one, …nah… six, enterprise servers/workstations on about $1.5k

ben — Sun, 16 Aug 2009 05:08:01 +0000

I have started building a relatively sophisticated new server. The objective was to have an extremely powerful, reliable, and expandable machine that could handle a broad cross section of tasks. All this without breaking the bank.

I first split up the tasks into a few broad categories. Included but not limited to:

Personal file storage (media, photos, isos, school and other historical files, etc) and sharing (nfs, samba, potentially streaming audio/video)
Data storage (scraped news in database, time series flat files, time series data files) and data processing (cpu intensive easily parallelized computations through compiled and interpreted code)
Development (programming + ide, svn, web server and testing environments) and deployment
Online backups (mozy — doesn’t support linux)
Personal computing (web browsing, Photoshop, video editing, audio/music editing and sequencing), although not much gaming
Testing random crap that I find online without screwing up the previous tasks via spyware, viruses, and general bloatware

Computers fail easily. They’re hard to upgrade, and it takes an enormous time to reconfigure a machine. This is still the case even if you migrate files over seamlessly, or maintain drive images. Because I didn’t want to deal with those kinds of issues, I opted to set up my server not only on new hardware, but do so entirely through virtual machines. The setup is roughly as follows:

Application/Persistent Server VM: http, svn, and other low CPU apps that need to be on 24/7. However, all files are NFSed from the:
File Server VM: Massive file server that has multiple virtual hard drives for various file types (arranged by category and access pattern; e.g. frequently used data files is one type, low use and bandwidth media files is another, svn repository is another).
Computation Server VM: Empty shell of a server to ssh into and run jobs.
Primary Linux Workstation VM: With the IDEs and apps I need like Eclipse
Primary Windows Workstation VM: Office, Photoshop, Mozy backups, etc
Scratch/Test Workstation VM: Well secured WinXP VM

I’m still in the process of setting all those machines up on my old desktop with VMware 6. There are a couple roadblocks I ran into:

64bit machines don’t always run nicely in a VMware, and require you to tweak with the processor Execution Mode. Often it helps to set the mode to binary translation and disable acceleration. I probably wont have a problem when on the actual server computer.
SVN hates the repository sitting on an NFS share. The solution, found after an hour of banging around, is to add “nolock” as an option for the nfs mount in /etc/fstab . Also, it’s important that usernames and ids are equivalent over the machines (e.g. the subversion user and group). Other than that, the guide at https://help.ubuntu.com/community/Subversion works nicely.

The next step is to deploy it on the hardware, which is arriving in the next week or so. The setup is as follows:

ASUS Z8NA-D6C Dual LGA 1366 Intel 5500 ATX Dual Intel Xeon 5500 Series Server/Workstation Motherboard
Intel Xeon E5504 Nehalem 2.0GHz 4 x 256KB L2 Cache 4MB L3 Cache LGA 1366 80W Quad-Core Server Processor (Starting with one, upgrading to 2)
Intel BXSTS100C Passive/active combination heat sink with removable fan (the cpu does not come with a fan!)
PQI 6GB (3 x 2GB) 240-Pin DDR3 SDRAM DDR3 1333 Desktop Memory Model MFC46GUOE-X3 (upgrading to 12GB soon. The board supports up to 6x8GB, but those 8GB chips mad expensive right now)
HITACHI 0A38016 1TB 7200 RPM 16MB Cache SATA 3.0Gb/s 3.5″ Internal Hard Drive (Starting with four of these in two 1TB Raid1 configuration. The host machine will run software raid under Ubuntu LTS server)
Athena Power BP-SATA3051B 3 x 5.25″ Bays to 5 x 3.5″ HD (SATA) Backplane (hot swap hard drive bays that also cool them down)
Also, a cheapo case+power supply, adapter/cables for eSata, and a CyberPower Home CP550SL 550VA 330Watts UPS.

Total: $1277. Add another processor for $230 to bring it to about $1500. Add 6GB more DDR3 ram to bring it to about $1600.

So we have…

Speed, upgradable: The Xeon processor is in the same league as the i7, and costs about the same as the i920. With the option for two on this board, you can get up to 8 cores (not hyperthreaded with the E5504. that requires the 5520). Seeing as the LGA1366 socket seems to be replacing LGA775, we might very well see an 8 core processor coming down the line soon. In that case, the computer can be upgraded to 16 cores. If that isn’t enough, the processing VM can be offloaded to another machine (or a cluster).
Disk space, backed up and upgradeable: The VMs are easy to resize and easy to relocate onto a new file server if needed. By sitting on a RAID that’s also offsite backed up, the data is pretty safe.
Stability and simplicity, since the various server functionality has been put into multiple VMs that are relatively easy to maintain independently.

There are some cons. First, the setup relies on proprietary VMware software, but other alternatives are available. Secondly, the performance is not going to be exactly like native code, and certainly graphics will be worse on this machine than one that has a native graphics card. Third, this setup is not built for perfect uptime. In fact, if the host fails, all underlying machines will fail. Luckily, data should be preserved, but this is not meant for production software/services. Finally, this setup takes quite a lot of time to get up and running… definitely not the right solution if a single low end non-vm dell/HP/etc server will satisfy your needs.

So if anybody is interested, I can keep you posted on how it goes.