Resume

Chris Stevens 12514 Culver Blvd Apt 11 Los Angeles, CA 90066 310-302-7480 chris@pwned.com

System Operations and Administration

Objectives

Experienced Systems Administrator with 30 years of dedication to system operations, seeking a challenging role where I can leverage my extensive technical expertise and passion for problem-solving to enhance system performance and user support. Committed to improving operational efficiency and providing exceptional service to foster a positive user experience in a dynamic IT environment.

Technologies

Linux, Containers, Kubernetes (Bare Metal), Switching, VLANs, LACP, DHCP, DNS, RAID, GitLab, SaltStack, PXE, Load Balancers, IPv4/IPv6, SMTP, Shell (Bash/ZSH) Dell Servers, iDRAC, Sphinx Docs, Fully Automatic Installation. MCP, CNA, CCNA

Work History

EdgeCast Networks/Verizon/Yahoo/Edgio Sr. Systems Administrator 2012 - 2024

  • Oversaw management and maintenance of 30,000 Linux servers across 200+ data centers worldwide

  • Over my 12 years of employment there, my team and I upgraded all servers from Ubuntu 10.04 to 22.04

  • Went through several generations of Dell servers as we upgraded and installed new POPs, from R510, R710, R720 R730’s, and R610, R620, R630 and R640’s. All servers were managed and provisioned remotely through iDRAC access

  • Deployed approximately 100 POPs with an average of 300 servers per POP

  • Maintained the lifecycle management of each server by making sure all firmware (iDRAC, BIOS, Lifecycle Controller, NIC’s, SSD’s etc.) was updated every time we provisioned or reprovisioned a server

  • Used fully automatic installation (fai-project.org) to build all of our servers via PXE booting. Once the OS was laid down, we’d use a configuration management tool to finalize the server configuration

  • SaltStack was introduced as a configuration management tool to replace dozens of shell scripts that configure our servers. Committed many changes to salt states and pillars as needed to make appropriate changes to the fleet

  • Decommissioned older and obsolete POPs to reduce operational expenditure or to create room for additional capacity, bandwidth, speed and next gen servers

  • Managed and maintained appdoximately 700 VMHosts and approcimately 2000 VM’s

  • Used Sphinx-Docs and GitLab CI/CD to build our documentation. This allowed us to migrate off of Confluence and inroduced the ability to use docs-as-code where we kept a git repo with all our documentation. While anyone was allowed to fork, clone, make a feature branch and send a merge request to the main documentation project, we enforced peer review requirements by requiring one approval from our US team and one from our London team. When the merge was approved the CI/CD process would build and publish the documentation to GitLab Pages

  • Used a validation system (smoke checks) that would continuously run and check the health of the servers. Smoke checks were simple bash or python scripts that ran every minute and sent alerts to the application owners of the servers. For example we’d check disk space, link speed, CPU utilization and application health and report issues through Slack or other means

  • Worked with the network team on a 80/20 basis for a couple months where I learned to establish peering arrangements with other ISPs also located at the same data center. At first we would peer via the data centers internet exchange (IX) and if the traffic grew to 1gbps or greater we would request to peer with them via Private Network Interconnect (PNI) where we would connect our router directly into theirs with a 10gb link

  • Prepared a course on how to use Kubernetes for my team, introducing them to the core components such as creating images, pods, deployments, services, configmaps, container registries and adjusting replica counts

  • As a team we took great pride in our work in helping others. We had a self-imposed 5 minute SLA for responding to people in Slack and were well known as the best place to go to for support

  • Maintained the Linux mail relays running Postfix

  • Worked with auditors on yearly PCI compliance projects

TASER International/AXON Sr. Systems Administrator 2009 - 2011

  • Managed and maintained several different server types (Linux, Microsoft) that ran the TASER’s evidence.com website. This website was used for uploading and displaying on-officer body camera footage from the AXON devices. Almost all videos of officer body cam footage that exist today are recorded from these AXON devices. These devices were showcased in the TV show “The Rookie”

  • One of my primary duties was a release engineer. I would receive software updates from our developers and deploy them to the production environment. This required taking a server out of the load balancer, perform the updates, then return the server to production by placing it back into the load balancer

  • As an operations team member, I was on-call 24/7 for any problems that may have required escalation

  • Often took trips to the Equinix data center to perform hands-on operations, such as replace bad hard drives, replace broken parts in servers or to be on standby during vendor-run system/firmware upgrades

Realtor.com Network Operation Center Technician Data Center Technician UNIX Systems Administrator 2004 - 2009

  • Spent a year as a NOC team member where I managed the health of the site using external venors as well as internal utilites such as Windows Perfmon. Escalated site/performance issues to the appropriate teams as necessary

  • As a Data Center Technician I was responsible for racking and stacking new Dell servers, Cisco switches & routers, creating custom length Cat 5 cables and generally making the data center and cabling look as clean clutter-free as possible. Troubleshoot Dell servers, working with Dell to receive replacement parts such as RAM, CPUs, RAID Controllers and hard drives

  • Maintained our forward facing DNS and SMTP environments, delivering 500k e-mails per day

  • Maintained infrastructure that ran the realtor.com, move.com, homestore.com and rentnet.com websites

  • Used NetApp NetCaches backed by a NetApp FAS-960 filer to store and deliver pictures of all listed properties on the websites, which numbered in the millions. For example, if we had 1 house listed on the site, and each house had 20 pictures, there were 5 different sized thumbnails of each picture created for display on different areas of the website which could add up to 100 pictures stored on disk for a single listing

  • Ran into a problem where through bad design we had too many pictures in one directory on our NetApps which would cause 30-second lag spikes. We diagnosed the problem with NetApp and worked with back-end developers to split these directories into smaller, easier to deal with “chunks” of files and directories. For example, a file at /vol/vol1/filename.jpg would be separated into deeper directories such as /vol/vol1/fi/le/filename.jpg

  • Swapped out 8 pairs of F5 BigIP load balancers down to 2 pairs of Citrix NetScalers which reduced our support cost significantly

  • Introduced a CDN into the environment so all pictures weren’t being delivered through our single datacenter. This sped up our picture delivery by seconds depending on your location

  • After introducing the Content Delivery Network (CDN) for our pictures, we then implemented a CDN-style Anycast DNS caching environment, where our primary DNS server was kept in-house while the secondaries were spread around the globe. In some cases, particularly where entries hadn’t been cached by resolvers, we saved seconds of time during initial DNS lookups

  • Part of team that built out and migrated our data center to a new location in Phoenix from Thousand Oaks. At this time we also moved to Dell Blade Chassis running VMWare and went mostly virtual. Made hundreds of custom length ethernet cables as well as serial cables which we used for the Lantronix devices we utilized for Out of Band management

e-MedSoft.com Systems Administrator 2003-2004

  • Implemented a telehealth solution that would allow doctors offices to communicate with each other over video using bonded ISDN lines

  • Later upgraded each site with a router (Cisco 2900 series) capable of using these ISDN lines on an on-demand basis, establishing the network links only as needed

  • Traveled to several states to setup and install the routers at remote locations

  • Performed daily tests to ensure systems were running smoothly

Beauty by Tova Systems Administrator 2000-2002

  • Upgraded old monochrome terminals connected to the RISC mainframe to PC’s that were capable of more than just running the basic sales system

  • Replaced all serial cables that connected the terminals to the Mainframe with Ethernet which saw a speed increas from 9600bps to 10mbps

  • Brought a T1 line into the office so the employees had internet access and could use our own website as they communicated with customers

  • Implemented Microsoft Exchange server 5.5 to bring e-mail to the company. One server in California and the other in New York.

  • Worked with vendors to implement a new telephone system so we could do more advanced things such as transfer calls and leave voicemails

OmniData Systems Engineer 1998-2000

  • Hired to be the full time technician for the Archdiocese of Los Angeles

  • Setup a Microsoft Exchange network running Microsoft Exchange 5.0/5.5 consisting of a central location in Los Angeles and 5 regional offices located between Santa Barbara and San Pedro

  • Churches that were involved in the Exchange network all dialed into their local regional offices, and the regional offices dialed into the central location to pass e-mail between themselves

  • All churches, regional offices and the central Exchange server were connected via modem using Microsoft Dynamic RAS connectors

  • Implemented hourly tests to make sure e-mail flow was uninterrupted and that the Central and Regional Exchange servers were functioning properly

  • When not working directly on projects for the Archdiocese, some of that time was spent building or troubleshooting computers for customers

Independent Computer Technology, Inc. Systems Administrator/Field Engineer 1994 - 1998

  • Managed service provider for many smaller companies that weren’t large enough to have their own IT department

  • Worked with various networking technologies such as LANtastic, Desk2Desk, Banyan Vines, Novell Netware (2.15 & 3.11)

  • Had a store front where we custom built PC’s and sold them to customers

  • Serviced and repaired broken PC’s that customers would bring into the store and offered on-site repairs as well

  • Ran a MajorBBS system that provided dial-up internet access to customers in the Simi Valley area and offered onlne multiplayer games, multi-user chat and message boards