JSFoo 2019

On component architecture, front-end engineering and Developer Experience (DX)

Tickets

Developing in a Large Monorepo

Submitted by Jai Santhosh (@jaisanthosh) on Sunday, 1 September 2019

Section: Full talk (40 mins) Technical level: Intermediate

View proposal in schedule

Abstract

In Microsoft, we have a very large TypeScript-based git repository where over 250 developers build and write code for high-value frontend components which is used across all Microsoft365 products like Outlook, Office, Bing, SharePoint, etc.It contains over a 100 npm packages, containing over a million lines of TypeScript code. Co-locating these components encourages collaboration and sharing code across teams very easier.

In this talk, we’ll focus on the tooling used and code organization to make the development easier, fast and reliable and will not discuss in detail unit tests and automation for a better focus on development.

Outline

A closer look at package management

Since there are 100+ highly inter-connected internal npm packages which declare over 2,500 external dependencies, we will briefly take a look at how packages and dependency management. We will discuss the structure of each package in the monorepo and how dependencies are expressed across these packages.

The packages contain transpiled TypeScript code and bundles generated using webpack and the modules from these bundles are used across various product endpoints.

Handling package dependencies

We use yarn workspaces to install our external dependencies and link the Midgard-hosted packages together based on the dependencies expressed in the package.json files. We will use an example with yarn workspaces to handle package dependencies in a sample monorepo.

Cross-package orchestration

The other major part of managing monorepos is cross-package orchestration. We would want to execute scripts across all the packages easily like build/packaging commands, linting, etc. We will take a look at achieve this using Lerna through the talk.

Using Test Apps to validate user behaviour

To validate more complex component behaviors, like user interactions, we use test-apps. Test apps are either webpages or native applications which load our bundles to render our components for manual testing. This is probably the second most popular validation workflow, therefore we care a lot about it.

Improving “watch” across packages

We will also briefly look at how we will manage watch processes for the monorepo-hosted packages. Its also important to discuss the Intellisense performance here.

We will go over a sample monorepo structure to explain this in detail.

Speaker bio

Jai Santhosh currently works on the Microsoft Search team at Microsoft. He previously worked on the Outlook Web Team and led some efforts on the mobile web version. He is quite passionate about the Web, JavaScript and is a Web perf enthusiast. He had previously led UI efforts at a fintech startup, ClearTax and worked on building Offline heavy web applications. He was also part of the first real-world Progressive Web App, Flipkart Lite and the new Flipkart Desktop website and had earlier worked in teams at large-scale like Yahoo! Maps and Media sites at Yahoo!.

Links

Slides

https://speakerdeck.com/jaisanth/developing-in-a-large-monorepo

Comments

  • Zainab Bawa (@zainabbawa) Reviewer a month ago

    Hello Jai,thanks for an interesting proposal. The implementation details are useful. The moot questions are:

    1. Why do you use this approach? What is the business and the engineering logic underlying this approach?
    2. Which other approaches did you consider/compare/benchmark before deciding on this approach?
    3. What is the one innovation of this approach that you consider a big win?
    4. What got compromised in the process of adopting this approach?

    Look forward to your responses.

    • Zainab Bawa (@zainabbawa) Reviewer a month ago

      Also, you may want to explain why Lerna was used, and what other alternatives was it compared with?

      • Jai Santhosh (@jaisanthosh) Proposer a month ago

        Why Lerna was used, and what other alternatives was it compared with?
        As mentioned earlier, we were use rushjs and gulp/custom scripts in other monorepos.
        Lerna was a library designed for optimizing the workflows around managing multi-package repositories.

    • Jai Santhosh (@jaisanthosh) Proposer a month ago (edited a month ago)

      Responses below:

      Why do you use this approach? What is the business and the engineering logic underlying this approach?
      We figured this was an efficient way to ensure each of the high-value component team can focus on their code in a
      monorepo which contains various other components as well. It is necessary to co-exist in the same repository
      but still keep it productive for every developer involved.

      Which other approaches did you consider/compare/benchmark before deciding on this approach?
      The other alternatives were rushjs (https://rushjs.io/), yarn workspaces, pnpm workspaces as well as we had a combination of gulp/custom scripts which
      were being used in other monorepos already.

      What is the one innovation of this approach that you consider a big win?
      Working with lerna and yarn workspaces allowed us to ensure each team’s package dependencies and overall package management was made easier.

      What got compromised in the process of adopting this approach?
      It took us a while to get to an efficient state and there is more to be done.

  • Zainab Bawa (@zainabbawa) Reviewer a month ago

    Thanks for an excellent rehearsal, Jai. Here are the points of feedback from today’s rehearsal, and which need to be incorporated in the next iteration:

    1. Start with explaining what is a Monorepo and why use it? What problems does Monorepo solve?
    2. Better still, start with a comparative example, where you show a sample Monorepo and compare building an app without a Monorepo. This will give a stronger start to the storyline of your presentation.
    3. From a people point of view, you have explained why you need a Monorepo. But from a codebase point of view, why do you need a Monorepo? This needs explaining.
    4. What are the pros and cons of using Monorepos?
    5. Need more information about drawback and the effort needed to set up Monorepo. How much does this cost – weigh this in. Also, your experience around build times, deployment side, and day-to-day issues. Want to see more content on this.
    6. Adding benefits to the systems should be explained (given the NPM modules, duplicity, YARN workspaces, etc). Otherwise the details are losing to the big picture which needs to be held together for the talk.
    7. Participants will want to see the Monorepo from an end-to-end app point of view. Give an idea of how these components are used across products. This will help to tie the story together.
    8. The title of the talk could be changed to something which is more about a story, rather than “Monorepo and TypeScript”. Participants should not get confused that Monorepo and TypeScript must go together.
    9. Explain that Google and Facebook use Monorepo also.
    10. There are two types of packages – one for consuming and one for publishing. When you build the graphs and charts, indicate which is what. This part is unclear.

    From a visual point of view, the following feedback has to be incorporated:

    1. Text on the slides is too small. Increase font size to 32 point.
    2. Some slides have too many bullet points. No more than 3 bullet points per slide.

    Incorporate the above feedback and share revised iteration by 22 September. We will schedule a final rehearsal next week.

  • Zainab Bawa (@zainabbawa) Reviewer 29 days ago

    The presentation was coherent and evolving nicely in the second rehearsal. The feedback points were as follows:

    1. What are controls and high value components? Is this a common way of representing? Or is this MS specific parlance?
    2. You mention serverless and cloud – is this a passing mention or will you elaborate on this?
    3. How does the check out process work for individual developers in Monorepos? Explain this in the slides.

    On the visual aspects of the presentation, use a different font than what is currently being used.

Login with Twitter or Google to leave a comment