Privacy Mode fellowship programme

Privacy Mode fellowship programme

Documenting privacy best practices in industry

S Kannan

Rohan Verma

@rhnvrm Author

Anish TP

Anish TP


Zerodha: Data protection, security and privacy practices

Submitted Aug 26, 2022


Zerodha provides an online platform in India to invest in stocks, derivatives, mutual funds, bonds, futures and options, and serves over 9+ million clients. In the financial services industry, it is mandatory to be compliant with the strict regulations that are enforced by multiple authorities. Zerodha has developed a solution to extend their existing microservices architecture and be fully compliant to meet the requirements.

Every internal microservices App is implemented for a specific domain, and Zerodha follows a zero trust stance strategy. The Apps use common libraries in the ecosystem, and Zerodha has extended the service architecture by implementing a centralised, distributed application, named Pigeon. The internal Apps place their user requests to message queues that are serviced by the Pigeon App. After fetching only the needed data from the Datasources, predefined templates are used to render the output. The Pigeon messenger workers are responsible to send this output to any SMS, email or push notification to end users.

The implementation of Pigeon follows many design principles, such as restricted access control, principle of least privileges, audit log retention policies, network segregation and data flow minimization. These best practices help keep up the highest design standards, meet the changing business needs, and also be compliant with data security and data privacy standards.


Zerodha provides an online platform in India to invest in stocks, derivatives, mutual funds, bonds, futures and options, and serves over 9+ million clients. In the financial services industry, compliance is a way of life and there is strict regulation enforced by multiple entities. It is mandatory to collect huge data including personal documents, scans and signatures. There are also requests for KYCs, Re-KYCs and Video-KYCs. It is thus essential to protect this sensitive data and adhere to security policies and regulations.

Problem statement

The standards and regulations change over time, and it is mandatory to implement the recommendations to continue to be compliant for business operations. The “Cyber Security & Cyber Resilience framework for Stock Brokers / Depository Participants” [1], for example, provides a list of the existing regulatory requirements. It states that “No person by virtue of rank or position should have any intrinsic right to access confidential data, applications, system resources or facilities. Stock Brokers / Depository Participants should implement an access policy which addresses strong password controls for users’ access to systems, applications, networks, and databases.”

The framework also states that “Stock Brokers / Depository Participants should ensure that records of user access to critical systems, wherever possible, are uniquely identified and logged for audit and review purposes. Such logs should be maintained and stored in a secure location for a time period not less than two (2) years.” Protecting user data and guarding against security breaches are very important in this industry.

As a heavily regulated ecosystem, often by multiple authorities, there are always tight deadlines to comply with the new regulations. The upcoming CERT-in (No. 20(3)/2022-CERT-In [2]) document, for example, calls for the reporting of cyber security incidents such as data leaks etc. to be reported to the authorities. Given an existing software architecture, Zerodha has been able to develop a solution to extend the design in order to be compliant with these regulations.

Software Architecture

Zerodha uses a microservices architecture where each microservice addresses a specific domain or problem set. They are neither huge monoliths nor are they tiny microservices, but they serve their purpose well. Each App uses standardised common libraries for HTTP responses, logging, pub/sub, response caching etc. and a stance of zero trust is used even for internal Apps. A zero trust stance is a strategic approach to cybersecurity that secures an organisation by eliminating implicit trust and continuously validating each and every interaction.

In order to meet the business and regulatory requirements, and to standardise the outgoing communication, a centralised, distributed App named, Pigeon was implemented. It provides an HTTP API to interact with the internal Apps, and this service communicates with messengers to send information to users via SMS, email or push notifications. It is based on a strict set of access control list-like permissions and allows internal apps to queue messages to this App, without having any direct access to sensitive user data. This greatly avoids any data leaks due to data duplication, or extraneous logging of client information at the application level. A high-level overview of the system is shown in the illustration.

![Illustration 1]("Illustration 1")

Design Principles

Principle of Least Privileges

The Principle of Least Privilege is the idea that any user, program, or process should have only the minimum privileges necessary to perform its function.

Zerodha’s internal microservice Apps use a client ID to perform any communication. There is no need to store any personally identifiable information such as name, email, phone number, address etc. Even the Pigeon application layer has only read-replica access tto the client database, and it uses a predefined template of limited variables for the information it fetches from the data sources. Thus, every resource uses only the data that it needs and can operate with the least privileges.

Access Control

Access controls restrict access to information based on the principle of least privilege. For example, the internal applications interact with Pigeon only through an HTTP API. Kong is an API gateway which enables restricting access to the Pigeon API to only applications for which valid tokens have been issued. Pigeon has user accounts that allow the user to authenticate themselves and gain tiered access to the Pigeon API. In the Pigeon application layer, strict network-level access control is enforced.

![Illustration 2]("Illustration 2")

Pigeon data sources are connected only to read replicas of the databases, and the connections are restricted with strict network access controls and username/passwords. Also, Pigeon messengers are only allowed to interact with upstream services that send the emails, SMS, or push notifications. Thus access controls are defined at various endpoints and interfaces across the systems and networks.

Audit Log Retention Strategies

The regulatory obligations require that logs be preserved for various audit purposes. The internal Apps are designed for specific domains, and hence may each have a different log retention period. Since the Apps are focused on a specific domain, we restrict any sensitive information to the App alone. All the App logs go to a data warehouse, where the logs are stored based on their retention policy. All the client communication is also masked. For example, the Kite Alerts App does not need to worry about logging the client email address, as it only uses the client ID, which is pseudo-anonymous (Example: XX1100, AB1234, etc).

![Illustration 3]("Illustration 3")

Network Segregation

Network segmentation involves segregating the network into logical units, thereby greatly reducing the attack surface. This segmentation restricts the potential damage of a compromise or security lapse to a single unit. At Zerodha, there are separate VPCs (logically, isolated, Virtual Private Clouds) for all logical units. No single unit can interact with another without requiring a proxy request, which is heavily restrictive, logged and audited.

Every employee requires a VPN (Virtual Private Network) and a zero-trust proxy system such as Pritunl for access. This, again, is for a specific amount of time only, and is strictly restricted to a set of employees. Also, a developer will have access only to the infrastructure that they manage, but not to one from a different business unit. There are no zero super users who have access to all the systems.

Data Flow Minimization

Data protection in transit is the protection of the data while it is travelling from network to network or from one application to the other. The amount of data in transit is greatly reduced when using the Pigeon ecosystem, since the applications consume only the data that is needed. Even the data provided by the data sources is templated such that only minimal data is transmitted across the network.

The Pigeon data at rest is of two types: Datasource data and Log data. Datasource data is managed securely as read-only replicas provided by the developer team for specific data. This is stored under encrypted disks and restricted access is provided through the Pigeon Datasource interface. For the logs, access is restricted to a set of individuals, and these logs are rotated based on their configured retention policies.

Reference Implementation

The Pigeon App is implemented using an asynchronous task queue Go library based on distributed message passing. It can work in either HTTP mode or as a worker. The internal Apps issue tasks that are queued to the Pigeon instance running in HTTP mode. The worker mode is responsible for processing these queued messages. This machinery uses Kafka brokers that can relay messages between the clients and the workers.

![Illustration 4]("Illustration 4")

Consider an example of pushing a message from an internal App to the user. A JSON request is sent by the App to the Pigeon API using the HTTP API with data source and messenger parameters as shown below:

![Illustration 5]("Illustration 5")

The message is queued and the internal App receives a message-id which can be used to query the message status. If any data source is mentioned in the request, the data is obtained from the read-only Datasource, and sent to the Messenger. The Messenger uses a predefined template with the fetched data, renders the output and sends it to the workers to push the notification.

An example of Pigeon composing the incoming data from the HTTP API into Datasource enrichment task, template rendition, and the messenger task is shown in the illustration:

![Illustration 6]("Illustration 6")

All the outward communication is logged and audited. Metabase dashboards allow querying the logs, and Rundeck is used for orchestrating the workers.


The Pigeon implementation provides a working example of extending an existing software design to meet the business needs. It is essential to protect sensitive user data, especially in the financial services industry. The design principles such as principle of least privileges, restricted access control, audit log retention policies, network segregation and data flow minimization provide the best practices for operating and complying with the necessary regulations, and also to keep up with the evolving changes to data security and data privacy standards.


  1. Cyber Security & Cyber Resilience framework for Stock Brokers/Depository Participants.

  2. N. 20(3)/2022-CERT-In Directions under sub-section (6) of section 70B of the Information Technology Act, 2000 relating to information security practices, procedure, prevention, response and reporting of cyber incidents for Safe & Trusted Internet.


{{ gettext('Login to leave a comment') }}

{{ gettext('Post a comment…') }}
{{ gettext('New comment') }}
{{ formTitle }}

{{ errorMsg }}

{{ gettext('No comments posted yet') }}

Hosted by

Deep dives into privacy and security, and understanding needs of the Indian tech ecosystem through guides, research, collaboration, events and conferences. Sponsors: Privacy Mode’s programmes are sponsored by: more

Supported by