Tuesday, December 22, 2015

Improvements to Uptime in Progress

Patchrends.com is the World's largest price database for Boy Scout patches, Order of the Arrow flaps, and many other pieces of Scouting's collectibles.  

Naturally, collecting thousands of price points everyday directly from eBay presents a number of interesting challenges. So does storing millions of database records and images. But, that's why I remain motivated to devote the time and energy into this invaluable service.

The Patchtrends.com Infrastructure

The service is divided up into several different pieces. The only piece that members and visitors get to see is the website. The website is hosted on a computer that is usually referred to as a "web server."  The other computers that support Patchtrends.com are focused on collecting data, organizing this data, and presenting a stable and reliable storage platform for the millions of data points and item images.

At the time of this writing, there is almost 1 TB (that's one thousand GB) of data collected. The vast majority of this data is made up of item images. And this is growing every day. Due to a lot of hard work, the site has room to grow into several terabytes of data before I have to consider expanding. Thankfully, storage mediums are cheap and open source software tools exist for making this backend scale nearly indefinitely. If there is interest in me expanding more on the technical aspects of the site and the infrastructure supporting it, I'd be more than happy to oblige.

Embracing the Cloud

As many of you may know, the webserver hosting Patchtrends.com has been at the mercy of flaky power at its colocation facility for some time now. I've been resistant in the past to pay additional fees for the hosting of the website itself, but I finally had enough of the stability issues the webserver was facing.

Recently, I took a major step in mitigating this major annoyance by migrating Patchtrends.com to a Digital Ocean, a provider of affordable virtual infrastructure. I looked at plans also for Amazon's EC2/AWS services, and determined Digital Ocean was a better fit for n ow.

While this move has not been without its technical challenges, it's almost mostly completed.  Much of the work was done to ensure that the webserver, once physically connected to the data collection and storage infrastructure, would be able to have fast access to the data no matter where the "server" was located.

This has been solved for the textual pricing data, but not yet for the millions of images that can't possibly be stored in the "cloud."  A solution is on the board, but it needs to be implemented. It's not all that involved, but will take some time to get right.

Goal: Zero Downtime

Zero downtime is not an unattainable goal for Patchtrends.com. In fact, it's very possible; and we've started down the path to achieving this state.

The end result of the move to the cloud is that the site is now much easier to replicate on a remote webserver. What is means rather is that I can build out multiple backup webservers that are able to mask the unlikely situation where any one of the servers becomes unavailable.

I am currently working on setting up a backup server through the services provided by Scaleway.com.  Whereas Digital Ocean is located in New York City (at least my instance is), Scaleway houses its infrastructure in Paris.  If the server is NY goes down for some reason (e.g., another Hurricane Sandy), then visitors will be directed to the Paris server without even knowing. Having more servers just makes our resiliency even greater.

And the additional servers don't just mean failover from unforeseen outages.  The same cluster of replicated servers can be used to balance out the load being put onto it from high periods of usage - a nice problem to have!

Patchtrends Needs Your Support 

Expanding our infrastructure by using paid services is a new expense for Patchtrends.com.  In order to help keep prices ridiculously low, please support my efforts by visiting the following eBay affiliate links presented for active items for sale at Patchbazaar.com.

In Conclusion

Patchtrends.com is poised to be ushered into modern capabilities - resiliency, high availability, and scalability. We've just taken a small, but critical step in this direction. Stay tuned for more!

WWW && YiS,

No comments:

Post a Comment