JSFoo 2019

On component architecture, front-end engineering and Developer Experience (DX)

Tickets

Developing in a large TypeScript Monorepo

Submitted by Jai Santhosh (@jaisanthosh) on Sunday, 1 September 2019

Section: Full talk (40 mins) Technical level: Intermediate

View proposal in schedule

Abstract

In Microsoft, we have a very large TypeScript-based git repository where over 250 developers build and write code for high-value frontend components which is used across all Microsoft365 products like Outlook, Office, Bing, SharePoint, etc.It contains over a 100 npm packages, containing over a million lines of TypeScript code. Co-locating these components encourages collaboration and sharing code across teams very easier.

In this talk, we’ll focus on the tooling used and code organization to make the development easier, fast and reliable and will not discuss in detail unit tests and automation for a better focus on development.

Outline

A closer look at package management

Since there are 100+ highly inter-connected internal npm packages which declare over 2,500 external dependencies, we will briefly take a look at how packages and dependency management. We will discuss the structure of each package in the monorepo and how dependencies are expressed across these packages. We will also look at the .tsconfig file organization to understand package dependencies internally in the monorepo.

The packages contain transpiled TypeScript code and bundles generated using webpack and the modules from these bundles are used across various product endpoints.

Using yarn workspaces to handle external dependencies

We use yarn workspaces to install our external dependencies and link the Midgard-hosted packages together based on the dependencies expressed in the package.json files.

Faster type-checking across these packages

The fastest validation is type-checking; when we produce invalid TypeScript code, IntelliSense gives us instant visual feedback within the editor. This is probably the most useful and loved validation step. So we want to make sure it works and it works fast.

Using Test Apps to validate user behaviour

To validate more complex component behaviors, like user interactions, we use test-apps. Test apps are either webpages or native applications which load our bundles to render our components for manual testing. This is probably the second most popular validation workflow, therefore we care a lot about it.

Using Lerna to prepare packages

TSC will produce the files described in the package.json file, so now the package is ready to be consumed. The *.js files are the JS files containing the code ready to be consumed. The *.d.ts files (DTS) are the files containing type information, they are used to type-check the code consuming this package. The *.d.ts.map are used by IntelliSense to enable cross-package code navigation. Because producing these files is necessary before a package can be consumed, we call this step “prepare”.

Because a package can be prepared only if its dependencies are already prepared, we need to run the prepare script in all packages in topological order. We use Lerna to orchestrate this.We usually use Lerna to run the watch process in each package of the dependency tree of the component we work on.

Improving “watch” across packages

We will also briefly look at how we will manage watch processes across the packages and optimizing by exposing packages pubic interfaces from JS+DTS files to TS files for the monorepo-hosted packages. Its also important to discuss the Intellisense performance here.

We will go over a sample monorepo structure to explain this in detail.

Speaker bio

Jai Santhosh currently works on the Microsoft Search team at Microsoft. He previously worked on the Outlook Web Team and led some efforts on the mobile web version. He is quite passionate about the Web, JavaScript and is a Web perf enthusiast. He had previously led UI efforts at a fintech startup, ClearTax and worked on building Offline heavy web applications. He was also part of the first real-world Progressive Web App, Flipkart Lite and the new Flipkart Desktop website and had earlier worked in teams at large-scale like Yahoo! Maps and Media sites at Yahoo!.

Links

Comments

  • Zainab Bawa (@zainabbawa) Reviewer 15 days ago

    Hello Jai,thanks for an interesting proposal. The implementation details are useful. The moot questions are:

    1. Why do you use this approach? What is the business and the engineering logic underlying this approach?
    2. Which other approaches did you consider/compare/benchmark before deciding on this approach?
    3. What is the one innovation of this approach that you consider a big win?
    4. What got compromised in the process of adopting this approach?

    Look forward to your responses.

    • Zainab Bawa (@zainabbawa) Reviewer 15 days ago

      Also, you may want to explain why Lerna was used, and what other alternatives was it compared with?

      • Jai Santhosh (@jaisanthosh) Proposer 4 days ago

        Why Lerna was used, and what other alternatives was it compared with?
        As mentioned earlier, we were use rushjs and gulp/custom scripts in other monorepos.
        Lerna was a library designed for optimizing the workflows around managing multi-package repositories.

    • Jai Santhosh (@jaisanthosh) Proposer 4 days ago (edited 3 days ago)

      Responses below:

      Why do you use this approach? What is the business and the engineering logic underlying this approach?
      We figured this was an efficient way to ensure each of the high-value component team can focus on their code in a
      monorepo which contains various other components as well. It is necessary to co-exist in the same repository
      but still keep it productive for every developer involved.

      Which other approaches did you consider/compare/benchmark before deciding on this approach?
      The other alternatives were rushjs (https://rushjs.io/), yarn workspaces, pnpm workspaces as well as we had a combination of gulp/custom scripts which
      were being used in other monorepos already.

      What is the one innovation of this approach that you consider a big win?
      Working with lerna and yarn workspaces allowed us to ensure each team’s package dependencies and overall package management was made easier.

      What got compromised in the process of adopting this approach?
      It took us a while to get to an efficient state and there is more to be done.

  • Zainab Bawa (@zainabbawa) Reviewer 3 days ago

    Thanks for an excellent rehearsal, Jai. Here are the points of feedback from today’s rehearsal, and which need to be incorporated in the next iteration:

    1. Start with explaining what is a Monorepo and why use it? What problems does Monorepo solve?
    2. Better still, start with a comparative example, where you show a sample Monorepo and compare building an app without a Monorepo. This will give a stronger start to the storyline of your presentation.
    3. From a people point of view, you have explained why you need a Monorepo. But from a codebase point of view, why do you need a Monorepo? This needs explaining.
    4. What are the pros and cons of using Monorepos?
    5. Need more information about drawback and the effort needed to set up Monorepo. How much does this cost – weigh this in. Also, your experience around build times, deployment side, and day-to-day issues. Want to see more content on this.
    6. Adding benefits to the systems should be explained (given the NPM modules, duplicity, YARN workspaces, etc). Otherwise the details are losing to the big picture which needs to be held together for the talk.
    7. Participants will want to see the Monorepo from an end-to-end app point of view. Give an idea of how these components are used across products. This will help to tie the story together.
    8. The title of the talk could be changed to something which is more about a story, rather than “Monorepo and TypeScript”. Participants should not get confused that Monorepo and TypeScript must go together.
    9. Explain that Google and Facebook use Monorepo also.
    10. There are two types of packages – one for consuming and one for publishing. When you build the graphs and charts, indicate which is what. This part is unclear.

    From a visual point of view, the following feedback has to be incorporated:

    1. Text on the slides is too small. Increase font size to 32 point.
    2. Some slides have too many bullet points. No more than 3 bullet points per slide.

    Incorporate the above feedback and share revised iteration by 22 September. We will schedule a final rehearsal next week.

Login with Twitter or Google to leave a comment