Padma was the new guy on the team, and that sucked. When you're the new guy, but you're not new to the field, there's this maddening combination of factors that can make onboarding rough: a combination of not knowing the product well enough to be efficient, but knowing your craft well enough to expect efficiency. After all, if you're a new intern, you can throw back general-purpose tutorials and feel like you're learning new things at least. When you're a senior trying to make sense of your new company's dizzying array of under-documented products? The only way to get that knowledge is by dragging people who are already efficient away from what they're doing to ask.
By the start of week 2, however, Padma knew enough to get his hands dirty with some smaller bug-fixes. By the end of it, he'd begun browsing the company bug tracker looking for more work on his own. That's when he came across this bug report that seemed rather urgent:
It had been in the tracker for a month. That could mean a lot of things, all of them opaque when you're new enough not to know anyone. Was it impossible to reproduce? Was it one of those reports thrown in by someone who liked to tamper with their test environment and blame things breaking on the coders? Was their survey product just low priority enough that they hadn't gotten around to fixing it? Which client was this for?
It took Padma a few hours to dig into it enough to get to the root of the problem. The repository for their survey product was stored in their private github, one of dozens of repositories with opaque names. He found the codename of the product, "Santiago," by reading older tickets filed against the same product, before someone had renamed the tag to "Survey Deluxe." There was a branch for every client, an empty Master branch, and a Development branch as the default; he reached back out to the reporter for the name of the client so he could pull up their branch. Of course they had a "clientname" branch, a "clientname-new," and a "clientname3.0," but after comparing merge histories, he eventually discovered the production code: in a totally different branch, after they had merged two clients' environments together for a joint venture. Of course.
But finally, he had the problem reproduced in his local dev environment. After an hour of digging through folders, he found the responsible code:
<li><a href="survey1.php">Survey #1</a></li>
<li><a href="survey2.php">Survey #2</a><span style="color:red">Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2)</span></li>
"But ... why?!" Padma growled at the screen.
"Oh, is that Santiago?" asked his neighbor, leaning over to see his screen. "Yeah, they requested a one-for-one conversion from their previous product. Warts and all. Seems they thought that was the name of the survey, and it was important that it be in red so they could find it easily enough."
Padma stared at the code in disbelief. After a long moment, he closed the editor and the browser, deleted the code from his hard drive, and closed the ticket "won't fix."
Continuously monitor your servers for configuration changes, and report when there's configuration drift. Get started with Otter