People used to think software development was “done” when the code was written and passed all its tests. But modern IT systems aren’t done until they are online, running, and integrated with their data feeds, storage, networks, and administration systems. This can involve very elaborate operations steps, such as dynamically creating a whole array of virtual servers, storage devices, accounts, software configurations, and network configurations.
These days, almost every company wants DevOps to automate such powerful infrastructure, speed up release cycles, and improve reliability and uptime. As an ultra-high-value and rapidly changing industry, Financial Technology lives near the cutting edge of innovation. Its DevOps can include continuous integration and continuous deployment (CI/CD), automated testing, containerization (as with Docker and Kubernetes), system monitoring, use of cloud features (like AWS, Azure, or a private cloud), virtual private clouds, extensive firewalls, advanced network security, and more.
FinTech IT projects have a lot at stake, and wise engineers will hand-pick DevOps priorities to match the project’s objectives and exposures. Let’s look first at what FinTech overall should expect from DevOps, and then at how different subfields should emphasize additional, specialized DevOps requirements.
What every FinTech solution needs from DevOps
Compared to other industries, FinTech places an unusually high priority on:
DevOps is maintainability’s best friend. As far back as 2013, here at FP Complete, we were releasing large upgrades to major server applications every few weeks. These days, at large Internet companies it is routine to see daily release cycles, and faster is quite doable.
FP Complete recommends a fully automated Continuous Integration and Continuous Deployment (CI/CD) system, including automated builds, an automated test suite, immutable containerized servers, and post-deployment health checks with rollback capability (Blue-Green deployment. If you haven’t implemented containers yet, you almost certainly should.
FP Complete also recommends a formal Quality Assurance system.
Data integration can be quite application-specific, but FP Complete recommends choosing a very small number of supported data formats, and having a clean layer providing these formats after data ingestion, normalization, and cleansing from a more diverse set of inputs. A service-oriented architecture (SOA) can make it easy to add new data-feed parsers in a completely language-independent manner, ensuring system extensibility. (See my comments on “Modular Design” here. Automated deployment and monitoring let you have more services running as separate processes, by eliminating the need to manually examine each server’s status constantly.
So DevOps can be a great help to FinTech in general. But there is still far more to be had — and our next priorities depend on which kind of FinTech solution we are building.
DevOps for Cryptocurrency
Cryptocurrency systems are of course sensitive to attacks in which a person attempts to steal the coins. Numerous real-world losses have been traced to a failure to implement proper DevOps, leaving opportunities for criminals.
If you’re implementing or trading a cryptocurrency, here are some DevOps issues to focus on right away:
Component isolation. To minimize the chances
of malfeasance, sensitive systems should be modular in design,
and unrelated components should run in separate processes —
ideally in completely separate VMs separated by firewalls. A
defect, code injection, privilege escalation, or social-engineering
attack on one service or component should still be unable to tamper
with another.If you are trusting your computer system to move and trade assets autonomously, you need absolute correctness, inviolable security, and very rapid response to trouble incidents. In addition to the concerns I listed for cryptocurrency , pay attention to these DevOps priorities:
|
Download our new guide |
This is a relatively forgiving application if your users are in-house or otherwise very tolerant of imperfection. (If you’re providing trading services to external parties, you can expect to be held to a very high standard.) Ask yourself, “what is the cost of a typical failure to our business?” Systems with a human in the loop are sometimes more error-tolerant than systems without a human in the loop.
However, you will need to think much more about usability testing, because a confusing UI update can introduce human error; and your automated test suite should include tests that drive the system through the UI, to detect coding bugs in that layer.
Human users of FinTech systems are often very powerful people, and a small number of unhappy users can make for a very bad day. So in addition to extensive testing, your DevOps practices may need to include gradual deployment of new versions to a test population of a few users (canary deployments), and support for both halts and rollbacks in case of significant trouble reports.
These tasks often amount to a medium amount of math, performed on a large number of input feeds and databases. Many firms construct unique asset valuation formulas by insightfully combining data that no one else was combining, or doing complex combinations with uniquely clever functions and formulas. Competitive advantage comes from generating unique insights, and these come from the ability to scale up innovative formulas and innovative data integrations quickly.
At FP Complete we regularly hear from FinTech firms that have built important analyses running on just one or a few desktops, who need these scaled up to a reliable server-based system. Beyond all the usual software engineering and cloud deployment problems, a key concern is maintainability . Analysts are used to updating their formulas several times per month and then sharing them with colleagues. And it’s important that new versions don’t always offline the old versions, which a colleague may still be using.
A version control system attached to a CI/CD system works wonders for safe maintainability. But it needs to be coupled to a simple metadata system that keeps track of which versions are now running at which addresses — and which allows versions with zero remaining clients to be shut down.
These applications require an exceptional amount of integration with legacy systems, some very old. They have to maintain an extremely consistent user interface, for use by clients who can be upset by the unnecessary change. They have all the requirements of an e-commerce application, such as resistance to sudden surges in demand. And they are subject to extremely large-scale Web-based security attacks, as the payoff for a successful criminal break-in could be enormous.
Voting, such as for shareholder votes or Board of Directors
elections, is a particularly sensitive subject, with huge decisions
being made and significant legal exposure. Ordinary voting is
rather similar to consumer banking and e-commerce (using votes
as the currency), but where governance rules require
anonymity, standards increase enormously. You must earn voter
confidence, and your systems should be able to pass a really
rigorous audit, including against insider malfeasance, while
protecting the privacy of each voter.
For such systems we recommend a DataOps solution in which raw inputs (from user interaction) are fully quarantined from the apps that handle persistent data storage, using very assertive firewalls and very low-permission accounts. An auditor must be able to verify that systems holding private user data are completely inaccessible from unauthorized locations.
Since the anonymization steps at the application layer may be intentionally irreversible, anonymized data should be stored with very high redundancy. It may be impossible to auditably reconstruct from scratch after identity data has been discarded.
For applications subject to extensive outside controls, it’s important to demonstrate adherence to the spec (application verification) and to be able to trace concerning behavior back from the running system to the code and checkins that caused it (traceability).
For verification, ensure that you have an automated test suite with organized test case management and that it is automatically fired up as a part of your CI/CD system. Be sure your full test suite is run before real deployments, not just your quick check test suite (sometimes called the smoke test) which is automatically run every time a build is done.
For traceability, ensure that your CI/CD system inserts serial numbers into your distributable software containers and other built artifacts, and records what artifact version numbers were used (including source code, libraries, and tools). And require that checkins to your version control system include links back to the requirements they were meant to satisfy.
To ensure that what went into production is still what’s in production, don’t grant permission to apply manual changes to running servers. Create admin accounts with limited permissions that can’t be used as “back doors,” and use immutable servers so that a new deployment is required when someone wants to change what’s on a server.
If your application is on the public Web, the server cloud design, the software maintenance schedule, and the network/firewall design all need to be designed to withstand malicious treatment. The average IT organization currently spends 12% to 15% of its budget on security.
DevOps can do much more to defend you than many people realize, and can make the most of your security budget.
Many security breaches happen through social engineering. Reduce these opportunities by automating control of your servers under distinct robot admin accounts, ones that normal users never use. Keep dangerous permissions away from regular IT staff going about their days.
Other critical breaches have famously happened through old, unpatched software with known vulnerabilities. Routinely audit the versions of operating systems and runtime components that are installed on all your servers, to ensure you don’t have obsolete ones in production. This can be largely automated.
Security breaches are worsened by having far too much access available in a single place, allowing a small intrusion to escalate into a big one. Take advantage of cloud network configurability , separate VMs and containers, and firewalls, to ensure that critical attack targets (like databases and production servers) are hard to reach and extra hard to enter as administrator, and to ensure that critical attack vectors (like front-end servers) are quarantined, firewalled, and monitored for unauthorized activity.
Unlike some older technologies, DevOps is not monolithic and can be implemented in small steps over an arbitrary period of time. Even the sequence of these steps is flexible. FP Complete recommends an incremental approach.
If you already have traditional software engineering and traditional system operations and monitoring in place, focus next on (A) streamlining your software engineering environment, or (B) containerizing and automating your deployment systems. Either makes a great next step and will put you well on the road to complete DevOps.
As always, remember that FP Complete is available to do a readiness assessment project for your DevOps, cloud, and other IT systems engineering. With experience on numerous advanced IT projects, we’re happy to team up with you with planning, design, implementation, knowledge transfer, audits, and upgrades.
Subscribe to our blog via email
Email subscriptions come from our Atom feed and are handled by Blogtrottr. You will only receive notifications of blog posts, and can unsubscribe any time.
