Global database – a big data analytics success
or how to manage high load via data streaming and custom indexing strategies
Global Database is a success story that started as an EBS Minimum Viable Product. The company grew into a business intelligence go-to solution. To date, it challenges giants such as ZoomInfo and DueDil. Its main goal: drive growth and minimize risks by offering data analytics technology for every enterprise. As a one-stop data shop, Global Database enables executives to identify high-probability prospects globally and run due diligence on potentials. It also aids account owners to keep their offerings relevant by engaging and monitoring existing stakeholders.
A Data Analytics Challenge tested via MVP
At first, project owners chased one goal: Implementing a digital service that allows stakeholders immediate data enrichment and global business insights.
“Back in 2015, we founded Global Database with a sole goal… Become the worlds’ largest source of private and public business intelligence. We sought to empower companies in reaching more customers by validating businesses, analysing markets, and enabling actionable data-driven decisions.” – states Mr Nicolae Buldumac, CEO @ Global Database.
To test the market, EBS Integrator chose a fitting technology stack. PHP for the app and Python at the data processing end. With a few thousand prospects, the solution held on nicely. However, supplying new functionality became troublesome. Furthermore, with the monolithic structure in place, scalability was somewhat expensive. One can only vertically scale so much. Since Global Database , an upgrade was paramount to keep up with set goals.
Designing for dynamic changes - a "must-have"
“To be honest, we mostly struggled with re-engineering data structures, flows and processing strategies. We , all to build a new data model designed for dynamic change.” – stated Nicolae Godina, the lead Business Logic Engineer who brought up Global Database.
Global Database growth led to quite the U-Turn. EBS Integrator had to go back to the drawing board and re-think their data analytics strategies. This of course, while accounting for complexities generated by feature enrichment. The former data model was good, but not good enough. In fact, this caused the U-turn. Everything demanded a re-build. Hence system architects and business logic engineers went back to the blueprints.
“Considering complexities, categories, and stereotypes involved in building the new data model, the new blueprint could no longer rely on a classic data layer architecture. To account for change, we had to migrate from traditional data structures to something that resembles a collection of data objects. Global Database had to evolve into a multi-platform system, or a service defined by loosely coupled micro-platforms.” – stated Ion Aremescu, the main System Architect that crafted the new system design.
A new system design where data analytics take precedence
After zooming in to each of the previous challenges, EBS Integrator designed a blueprint, where data analytics fit the users’ needs, as opposed to forcing stakeholders into pre-defined models. The team went from monolithic to a microservice paradigm. Docker replaced the old virtualization means. A “Kubernetes as a Service” approach resulted in fewer DevOps dependencies, enabling a seamless CI/CD environment. ReactJS powers the front-end, while Java on Spring replaced most of those Python data-critical modules. Since the new architecture resembles a loosely coupled web of services, EBS dropped any programming language dependencies. Hence, Python use-cases that kept their relevancy, still power some platform elements.
“Before going this route, processing data arrays could take up to a minute. Now, GDB’s data processing metrics fell under 20 seconds per high-load queries and in some instances, these times go as low as 150 milliseconds.” notes Nicolae Godina, Global Databases’ main Business Logic Engineer.
Data Analytics as a Service powered by Kafka
At the centre of Global Database’s competitiveness lies the essential: crispy-fresh public data. Prior to the upgrade, this challenge alone relied on a heavy operational load and hundreds of Python structures. A data update was likely to interrupt service delivery. This could’ve been acceptable for an MVP, but not for the fresh build.
Here, the entire data analytics, enrichment and transformation approaches were refreshed. Operations handling enrichment and fetching were built under the data streaming paradigm, powered mainly by Kafka. Batch processing hasn’t been cut off. Instead, these approaches have been reutilized to ensure consistency where required and reduced inconsistency risks. EBS also re-engineered the way analytics are fetched to the users’ benefit.
“What EBS did has certainly exceeded all our initial expectations. We’ve seen an impressive drop in server resource usage, an UX that is faster, snappier, more relatable to our end-users and we haven’t had an outage in years.” – states Mr Nicolae Buldumac, CEO @ Global Database.
SQL, NoSQL, and ingenious indexing a solution to efficient data analytics fetching
If you’re into the data business, Elastic Search, are stack elements that aren’t a foreign term. You’re well aware in some instances, Elastic Search ain’t that elastic and in some instances, betting on NoSQL ain’t the best practice. Our team has long-learned that the best fit lies in the middle: a combination of both relational is a fit, especially when one complements the other. Accounting for all complexities at the system architecture edge, EBS Integrator developed several indexing strategies that cut data fetching times by 80% to the users’ benefit. As a result, all interfaces became snappier and compiling reports takes milliseconds.
“To solve the most critical GDB challenges we’ve started with the way data is processed. Since our new infrastructure made use of Kafka, we could change the entire data paradigm and enable an asynchronous . As a result, query times have been reduced with 80% building premises for a healthier and more reliable processing environment. “ – states Mr Ion Aremescu Global Database’s main System Architect
Data Analytics that fit client tools as a glove
Moving away from the MVP to a top-performing alternative was just not enough. As Global Database grew, so did its partner list. This outlined a new enrichment challenge: building enrichment services that adapt to partner APIs and client third-party analytics systems. To solve this dilemma, EBS Integrator engineered a unique ETL platform, composed of scalable services that can translate any API request or data stream into a common language for both the main system and client’s environment.
As a result, adding a new partner data stream or support for external tools does not require interventions on the live system, but rather building yet another service/outlet within the ETL platform, slashing development, testing, and deployment times in half. “We’ve decided to make Global Database a universal fit when integrating with third-party systems or adding another enrichment platform to our ecosystem. It acts as a synchronous translator for 3rd party . If there’s a foreign language to add, all we have to do is add a new vocabulary and grammar instruction and we have a go.” – noted Sergiu Botez, Global Database’s Product Owner.
Operating on a live data analytics service
No matter how flexible the new system design is, upgrading a live service remains a conundrum. “Given our customer’s circumstances, we knew: time is of the essence. The entire development cycle could take over 6 months and by the time the new version of GDB would emerge, their stakeholders might have reached for a better service to their competition.” – noted Mr Ion Aremescu.
Once set in motion, the upgrade focused on migrating the MVP to a container-based infrastructure gradually. The shift took place synchronously for all concepts: data, infrastructure and operations in the background, unseen to active users. The gradual migration advantage enabled EBS to maintain an online service while migrating key components to microservice structures.
The change in key results
The resulting platform didn’t meet, it exceeded, expectations. To date, Global Database handles processing and aggregates over 50 million companies. That includes historic, financial and “good-faith” scores, employees, previous engagements, as well as active updates as per each company related record. “We’re working with more than 130M companies and we are planning to increase by 2x this number in the next year. Our platform has now 4 different applications for various customers’ needs, such as sales, purchasing and acquisition, compliance, and marketing.” – notes Sergiu Botez, Global Database’s Product Owner.
Processing bigger data analytics faster and more reliably
With constant updates in motion, in less than two years, Global Databases’ weight shifted from Gigabytes to Terabytes. The evolution was obvious and with each data partner, the scope was bound to get even bigger. Fortunately, the platform’s data model and infrastructure have been designed around that, and today, Global Database is ready to process Petabytes. “GDB handled hundreds of gigabytes, processing bottlenecks were starting to cripple our service. Before the upgrade, processing Gigabytes would have been an issue, but with the benchmark figures we’ve seen on the new platform, we’re ready to take on our first Petabyte. – states Mr Nicolae Buldumac, Managing Director, at Global Database.
Accelerated time-to-market figures
At EBS CI/CD is no longer a luxury, but a requirement, especially for partners such as Global Database. All of the system architecture decisions lead to developing a multi-sided platform with no expiration date. Since DevOps follows a “Kubernetes as a Service” approach, developing new features take days instead of weeks, weeks instead of months and months, instead of years.
“The main objective here was to build a scalable architecture, that runs in the cloud, not because it’s fancy – we’ve designed it this way so Global Database could leap ahead of its competitors. The approach might not have been perfect but based on NPS and benchmarks, I figure we’ve got pretty close”. notes Nicolae Godina, Global Databases’ main Business Logic Engineer.
Global Database standing on its own - from a client to a partner paradigm
This success story isn’t just about technical excellence alone – it’s about growing Global Database as a solution provider. “At EBS locked-in practices aren’t acceptable. Instead, we’re guiding each of our clients throughout a comprehensive piloting program, so they can maintain and expand their service on their own if they wish to do so. With Global Database, we went the extra mile. Using our know-how, internal resources, and best practices, we build Global Database’s own technical division.” – stated Mihai Tugui, EBS Integrator’s CTO.
In less than 6 months, EBS Integrator grew an entire IT department for Global Database that handles all platform development. EBS resources have swapped their roles, from delivery to mentorship, still enabling Global Database to reach its #NextLevel.“What EBS did has certainly exceeded all our initial expectations. Our platform now runs for 5.000+ happy customers that can prospect through 130M companies and 150M contacts; our public website is visited by over 17.000 potential customers daily and we’re able to expand our service without any constraints.” – states Mr. Nicolae Buldumac, Managing Director, at Global Database.
Get in touch to find out more about how
our experts can help your business