BI, Analytics and Data Engineering professional with 20+ years of software industry experience.
I am a technical leader with 20 years of experience in building and running software products for various domains. I lead the Business Intelligence and Analytics Data Engineering function at Beat Mobility. Until very recently, I worked at Omio in Berlin where I led a team of data and analytics engineers to build a strong BI and data engineering foundation. My team and I owned the data platform that fueled the growing needs for business intelligence at Omio and supplied reliable, trustworthy data to multiple stakeholders. Some of my recent achievements include (a) Created a Business Intelligence / data-platform evolution strategy for Omio and grew the platform from its nascent stage into a sophisticated, event-based, self-service big data platform. This platform now covers multiple data-sources, ingestion systems, storage (data warehouse and lake) as well as analytics/visualization services. (b) Set up systems and processes to enable data-ops and self-servability of analytics at Omio and reduced dependence on central data teams. (c) Developed strong relationships with multiple stakeholders across the organization to provide reliable data products for their rapidly growing needs. (d) Led the growth and evolution of engineering systems and architecture, successfully managing tech debt, future road-map and rapid prototyping to keep Omio's data engineering systems up-to-date. (e) Grew a motivated team of engineers and analysts with active participation in their career development. (f) Invested in improving the ROI from data by through multiple initiatives to improve the data culture at Omio. These include setting up a community of practice (Data Guild), introducing tools like dbt to improve Analytics workflow and setting-up team topologies to improve productivity and extract the latent value in data. (g) Successfully established data privacy and GDPR processes within the scope of the analytics data org. I am an engineer at heart and fairly hands-on with coding. I am also active in many open source communities in the data space. Outside work, I am passionate about giving back to the community by mentoring budding engineers and engineering managers across the world. I offer free mentoring over hangouts as a member of a mentoring club. Occasionally, I provide consulting to startups in Europe and India on their data engineering setups. I write frequently on data, strategy, engineering and leadership. Blog: https://rahulj51.github.io/ Twitter: https://twitter.com/rahulj51 Mentoring Club: https://www.mentoring-club.com/the-mentors/rahul-jain
Ask me about
Jun 2021 - Present
Head of BI and Data Engineering
Apr 2020 - Present
Mentoring and coaching engineers and new engineering leaders on varied topics - https://www.mentoring-club.com/the-mentors/rahul-jain
Nov 2017 - Jun 2021
Principal Engineering Manager, Business Intelligence
Key accomplishments: (a) Evolved the data engineering architecture from an early stage batch heavy system to a state of the art data platform based on real time streaming systems. (b) Stabilized the data pipelines from an MTBF of 10 days to 90 days. (c) Significantly improved timeliness, quality and reliability of analytics data. (d) Led the development of self serve, real-time data ingestion platform based on Kafka and Apache Spark that enabled any team at Omio to setup their own ETL pipeline to send data to warehouse. (e) Grew the BI engineering team from 4 members to 10. (f) Executed a complex data warehouse migration project to move from Redshift to BigQuery. This migration resulted in an annual saving of $200K in cloud cost. Responsibilities included (a) Enabling the data engineering team meet their short and long term goals. (b) Providing a vision to the team for the evolution of our data platform and data strategy. (c) Providing technical leadership to my team with respect to system design and architecture, managing tech debt and future planning and prototyping to keep our engineering systems up-to-date. (d) Working with multiple stakeholders including BI analysts, data scientists and other business functions to provide data for their rapidly growing needs. (e) Working with the engineering leadership to establish technical best practices in the area of data engineering (f) Growing a motivated team with regular feedback cycles and participation in their career development. (g) Managing data infrastructure cost (h) Managing data governance around privacy (GDPR), meta-data management and data quality management.
Sep 2012 - Nov 2017
Tech Lead and Lead Consultant
Key responsibilities include: Technical Leadership Responsible for end to end delivery of product features Architecture and Design of software systems Implementing agile technical practices and processes Agile Consulting Mentoring teams Currently helping a client of Thoughtworks create a pricing decision making system that recommends pricing strategies and will help pricing managers in retail organisations make informed decisions about their pricing strategies. As the technical lead for this product, I am involved in - Architecture and design decisions for the product using Big-data toolset (HDFS/Spark/ElasticSearch). The product ingests and processes very high volume data (1 few hundred million rows per week). - Performance engineering and tuning of ElasticSearch cluster for approx. a few hundred million writes (ETL) and complex reads. - Development and support of the platform's various business processes as micro-services (using Golang and Scala) - Developing the front-end of this product using React.js/Redux Recent Experience * Creating a set of RESTful APIs (Dropwizard/Jersey/Mulesoft/Moodle) to allow external e-learning systems to integrate with a learning managament portal. * Enhanced a data-ingest pipeline for a learning management portal by replacing ActiveMQ with Apache Kafka and Apache Camel for better reliability and performance. The pipeline manages asynchronous processes like search-indexing and transformation for the portal. * Created a set of Restful APIs for a very large US based travel portal. These APIs allowed advertising partners of the client to download various reports of financial and commercial value. This application was built on top of the client’s Apache Hadoop platform (using Apache Hive and Oozie workflows) and integrated tightly with other information systems used by them.
Jul 2007 - Sep 2012
Senior Analyst and Java Development Lead
I was mainly responsible for writing/enhancing/maintaining a plethora of java/j2ee based processes used for ingesting large volumes of financial market data (balance sheets, stock prices, earnings estimates etc.) into our systems and developing several downstream processes (applications, reporting frameworks, ETL tools etc.) that use this data. I also developed web based dashboard to help our production support team monitor all production processes from a single platform.
Jan 2000 - Jun 2007
Software Developer and Project Lead
Worked on various projects, big and small, as a senior developer and later as a project lead. Handled multiple responsibilities including - Software development and programming - Design - Testing/Debugging - Performance testing - Onsite coordination at client site - Technical Leadership
altMBA45 Ogre Cohort, Leadership
1995 - 1999
Indian Institute of Technology (Indian School of Mines), Dhanbad
Bachelor of Technology (B.Tech.)